The quote workflow works. It survives crashes, waits for the underwriter, contains failures and leaves one trace per run. Then the insurer grows. Instead of one run at a time, there are thousands a day. Instead of three agents in a fixed pipeline, a second team proposes peer agents that consult each other: the assessment agent may ask the underwriting agent for a preliminary view, and the underwriting agent may ask the assessment agent for more evidence. A third team wants to shave seconds off the broker's wait by starting the draft before all the evidence is in.
Each proposal is reasonable. Each introduces a problem that did not exist on the earlier pages. The draft started early is sometimes wasted. Two agents that consult each other can end up each waiting for the other. A queue that always serves the busiest broker first leaves a small broker's request waiting all afternoon. And when the recommendation is wrong, there is no single agent to blame, or to measure.
This page is the expert layer of the site. The example implements only part of what follows; each section says what ran and what is a description. One note before any output: the agents here are scripted stand-ins and time is simulated, so these results show how the orchestration works, not how good or how fast any model is.
1. Speculative execution: start before you know#
Speculative execution means starting work before it is known to be needed, so that if it is needed, it is already done. Two forms matter in an agent workflow.
The first is a hedged call. The roof lookup usually answers in 180 simulated milliseconds and sometimes times out after 2,000. Instead of waiting for the timeout and then retrying, the orchestrator starts a second, identical call after a short wait (say 500 ms) if the first has not answered, and uses whichever answers first. With the example's illustrative durations, Monday's roof lookup (first attempt times out, second answers) would finish at about 680 ms instead of 2,680 ms. That number is arithmetic on the constants in core.py, not a run: the example does not implement hedging.
The second form is speculative drafting. The underwriting agent starts its draft as soon as the roof assessment and the rules are in, before the slower guidance search returns. If the guidance changes nothing, the draft is ready earlier. If the guidance changes the outcome, the draft is thrown away and written again.
Both forms trade cost for time, and the trade has rules:
| Rule | Why, in the quote workflow |
|---|---|
| Only speculate on work without side effects | A hedged roof lookup is a read; a hedged record_recommendation would be a second write, unless idempotent (see durable workflows) |
| Never let a speculative result skip a check | A draft started early still passes the guard in step 5, against the evidence that finally arrived |
| Cancel the loser | The slower hedged call should be cancelled, or at least its result ignored and its cost counted |
| Count the waste | Every discarded draft is a model call paid for; the budget must include speculative work |
| Speculate where the guess is usually right | Drafting before guidance arrives pays only if the guidance rarely changes the outcome |
Who decides here. Code decides when to speculate and which result wins, by a rule fixed in advance. A model may produce the speculative draft, but it never decides whether its own early draft is good enough to keep. The guard does, against the final evidence.
For a backend engineer. A hedged request is the same idea as sending a read to two replicas and taking the first answer. Speculative drafting is closer to branch prediction in a CPU: the work is done on a guess, and the guess is checked before anything is committed. Where the comparison stops: a CPU's wrong guess costs a few cycles; a wrong speculative draft costs a model call, every time.
2. Deadlock: everyone waits for someone#
A deadlock is a state in which two or more parts each wait for another part of the same group, so none of them can ever continue. The classic form is a cycle: A waits for B, and B waits for A.
The peer proposal creates exactly that risk. The assessment agent asks the underwriting agent for a preliminary view before deciding which evidence to gather. The underwriting agent, by design, does not draft without the evidence package, so it asks the assessment agent for evidence first. Each holds its question open and waits. No error is raised; both runs are "in progress". The broker waits until someone notices.
The example does not implement a cycle between agents. It does show the simpler relative of a deadlock, a stall, in its peer topology: when the assessment agent never answers, the intake agent has handed off and stopped, and nothing else is waiting to notice. The simulation, run in a scratch copy:
import topology
r = topology.simulate_topology("peers", "escalate", "assessment agent")
print(r["status"])
print(r["outcome"])
stalled
The intake agent handed off and stopped; the assessment agent never answered. No agent owns the run, so it stays stalled until a timeout outside the agents finds it.
The four conditions under which a deadlock can happen were described for operating systems long before agents: parts hold something while waiting for something else, nothing can be taken away from a part, waiting is allowed, and the waits form a cycle. Removing any one of them prevents deadlock. In agent workflows the practical removals are these:
| Remove | How, in an agent workflow |
|---|---|
| The cycle | A fixed direction of requests: the assessment agent may ask the underwriting agent nothing; only the orchestrator routes back |
| Waiting without limit | Every wait has a timeout owned by the orchestrator, not by the waiting agent |
| Holding while waiting | An agent returns what it has (with missing named) instead of holding the case open |
The quote workflow in pipeline mode cannot deadlock: its steps form a fixed order with no cycle, and the one wait (for the underwriter) has a timeout that escalates. That is a structural guarantee, which is one of the strongest arguments for the hybrid design on control plane.
Diagram: Three ways to stop making progress · and the structural fix for each
Read left to right. A deadlock is a cycle of waits: nothing moves. A livelock is a cycle of actions: everything moves, nothing changes. Starvation is a queue order that never reaches some work. Each has a fix that lives in the orchestrator's code, not in an agent's instructions.
- 1 · Deadlock
- A waits for B, B for A: The assessment agent waits for a preliminary view; the underwriting agent waits for the evidence. Neither can continue.
- One direction: Requests between agents go one way; every wait has a limit owned by the orchestrator.
- 2 · Livelock
- Back and forth: Each handoff succeeds; the case returns again and again with one more request.
- Budget across agents: A limit on handoffs per run, and a rule in code for what happens when it is reached.
- 3 · Starvation
- Never chosen: Other work is always picked first, so some work waits indefinitely while the system runs.
- Fairness policy: A written rule for which work runs next when the pool is full, for runs and for people's work lists.
3. Livelock: everyone moves, nobody progresses#
A livelock is the busy cousin of deadlock: the parts keep acting, each in response to the others, and the system still makes no progress. Two people stepping aside for each other in a corridor, in the same direction, again and again.
In agent systems livelock usually looks like politeness or caution. The assessment agent hands the case to the underwriting agent; the underwriting agent finds the evidence incomplete and hands it back with a request; the assessment agent gathers one more passage and hands it over again; the underwriting agent asks for one more. Each handoff succeeds. Each agent is doing what its instructions say. UW-904 shows the one-agent version: the stand-in searches the guidance again and again with new wording, every call succeeds, and only the budget ends it (see limits).
Livelock is harder to detect than deadlock because there is no silence to notice. The defenses are the ones from limits, applied to the whole system: a budget across all agents of a run (handoffs, not only turns), a count of how often the same pair of agents has handed the case back and forth, and a rule in code for what happens when the count is exceeded. In the quote workflow that rule already exists: escalate to a senior underwriter with the reason.
Common confusion. Livelock is not a slow run. A slow run makes progress at a low rate; a livelocked run makes none, however fast it moves. The test is whether the state that matters (the evidence, the draft) changes between rounds.
4. Fairness and starvation#
Now the queue. Thousands of runs share a pool of workers and a roof provider's rate limit. Most come from a few large brokers who send batches; some come from small brokers who send one request a week.
Starvation happens when some work waits indefinitely because other work is always chosen first, although the system as a whole keeps running. A queue ordered by "whichever broker has the most pending requests" starves the small broker. So does a queue where retries jump ahead of first attempts during an outage.
The example shows starvation at the scale of one step. On the concurrency page, Monday's three lookups run on one worker with a 3,000 ms deadline. The roof lookup and its retry come first in the queue, the slow guidance search takes the worker until after the deadline, and similar_decisions never starts at all. Nothing failed. The lookup was starved by the order of the queue and the size of the pool.
Fairness is the property that every piece of work gets a share of the resources and eventually runs. It is a policy, written in code, and there are several:
| Policy | What it guarantees | What it costs |
|---|---|---|
| First in, first out | Nothing waits behind work that arrived later | A burst from one broker delays everyone behind it |
| Round robin per broker | Each broker gets a turn | A broker with one urgent case waits for others' turns |
| Weighted shares | Each broker gets a fixed share of workers | Shares must be configured and reviewed |
| Aging | Waiting work gains priority over time | More bookkeeping; a priority can still be gamed |
Fairness interacts with the human wait. A run suspended for an underwriter holds no worker (see human approvals), so it does not starve others. But the underwriters themselves are a queue, and the same policies apply to their work list: an escalated case that always sorts below fresh cases is a starved case with a person in the loop.
5. Evaluating many agents#
The last problem is measurement. In a single-agent workflow, a wrong recommendation has one author. In a multi-agent workflow the intake agent extracted the facts, the assessment agent chose the evidence, the underwriting agent wrote the draft, and code decided the order. When the outcome is wrong, which part failed? And when you change one agent, how do you know the system improved?
Multi-agent evaluation judges a system of several agents on the outcome of whole runs, then attributes failures to parts. Four questions, in order:
| Question | How, with this site's tools |
|---|---|
| Is the end state right? | Outcome evaluation per run (see tracing), over many trials |
| Did the required process happen? | Invariants and the guard: eligibility ran, cited rules were retrieved, a person decided |
| What did it cost? | Model calls, tool calls, tokens, time per run, from the trace |
| How does it behave when one part fails? | The same runs with one agent down or slow (fault injection, see testing) |
The topology simulation of the example answers the third and fourth questions for three arrangements on the same case. It ran in a scratch copy:
import topology
print(f"{'topology':11} {'agent down':17} {'status':10} model calls")
for t in topology.TOPOLOGIES:
for down in (None, "assessment agent"):
r = topology.simulate_topology(t, "escalate", down)
print(f"{t:11} {down or '-':17} {r['status']:10} {r['model_calls']}")
topology agent down status model calls
router - suspended 4
router assessment agent failed 2
supervisor - suspended 7
supervisor assessment agent suspended 5
peers - suspended 3
peers assessment agent stalled 1
With every agent healthy, all three reach the same state: suspended, waiting for the senior underwriter. On outcome alone they are equal. They differ on cost (3, 4 or 7 model calls in this simulation) and, more sharply, on behavior when the assessment agent is down: the router fails visibly, the supervisor routes around the missing agent and still suspends for a person, and the peers stall with no owner. An evaluation that only scored healthy runs would rank them by cost and pick the one that stalls. The numbers are counts in a scripted simulation, not measurements of any model or product.
Attribution comes after. The trace says which agent produced what; a counterfactual run says which part caused the error: replace one agent's output with the correct one from the case file, run again, and see whether the outcome becomes right (see the folded section on tracing). Doing this over many failed trials gives a table of failures by agent, which is where improvement work should start. Measuring rates over many trials, with intervals and release gates, is the subject of Eval Engineering.
What this is not. This page names the problems that appear when many runs and many agents share a system. Operational guarantees at that scale (service levels, load shedding, incident response) belong to Agent Reliability Engineering; the messages agents exchange and the A2A protocol belong to Agent Communication.
The same idea in a coding agent. Several coding agents working in one repository meet every problem on this page. Two agents editing the same file wait for each other's branch to merge (a deadlock if each needs the other's change first). Two agents that fix and re-break each other's tests livelock. A merge queue ordered by size starves small pull requests. And when a release breaks, the question "which agent's change caused it?" needs exactly the counterfactual method above: revert one change, run CI again. Merge queues and CI are covered on Diff to Deploy.
Who decides here#
| Decision | Who |
|---|---|
| When to speculate, and which result wins | Code, by a fixed rule; the guard checks the winner |
| Which agent may ask which (the direction of requests) | Code: the topology, fixed at design time |
| When a wait or a back-and-forth must end | Code: timeouts and budgets owned by the orchestrator |
| Which work runs next when the pool is full | Code: the fairness policy of the queue |
| Which arrangement of agents to ship | People, from an evaluation on outcome, process, cost and behavior under failure |
| What a model decides | Content inside its step; never the order of the queue, a timeout, or its own evaluation |
Checklist#
- Speculation is limited to work without side effects, and every speculative result still passes the guard.
- Speculative work is counted in the budget, and its waste is measured.
- Requests between agents have a fixed direction; no two agents can wait on each other.
- Every wait and every back-and-forth between agents has a limit owned by the orchestrator.
- The work queue has a written fairness policy, and the oldest waiting run is monitored.
- The underwriters' work list has a fairness policy too, so escalations do not starve.
- An evaluation of the system scores outcome, process, cost and behavior with one agent down.
Detecting a deadlock in a running systemWait-for graphs, timeouts and who breaks the cycle
Prevention by design is best, but a system that allows agents to call each other needs detection too. The classic method is a wait-for graph: a node per agent invocation, an edge from each waiting invocation to the one it waits for, and a periodic search for a cycle. The orchestrator can build it because every request between agents passes through it (another reason to keep routing in the control plane).
When a cycle is found, someone must break it. Choosing a victim (the youngest invocation, the cheapest to redo) and ending its wait with a "could not complete" result lets the others continue with missing named. In practice, most teams rely on timeouts instead of detection: simpler, and good enough when waits are rare. The cost is that a deadlock is discovered only when the timeout fires, and is reported as a timeout, so the trace must record what each part was waiting for.
Non-determinism makes these problems intermittentWhy a deadlock may appear once in a thousand runs
With deterministic agents, a deadlock either always happens on a given input or never does, and a test finds it. With models, an agent asks for more evidence in some trials and not in others (UW-010: nine paths in twenty trials, in a scripted stand-in). A cycle that needs both agents to ask at the same moment appears in a small share of runs, under load, and disappears when someone tries to reproduce it.
This is the strongest argument for structural guarantees over behavioral ones. "The agents usually do not ask each other" is a property of the model on today's inputs. "The assessment agent cannot address the underwriting agent" is a property of the code. Only the second one survives a model upgrade.
Evaluating a change to one agentLocal improvement, system regression
Improving one agent can make the system worse. A more thorough assessment agent gathers more evidence, which makes the handoff larger, which makes the underwriting agent's drafts slower and more expensive, and occasionally confuses it with passages it did not need (see handoffs). An evaluation of the assessment agent alone shows a gain; an evaluation of whole runs shows a loss.
So a change to one agent is evaluated twice: locally (does this agent do its part better, on its own inputs?) and on the system (do whole runs end better, at what cost, with what behavior under failure?). Ship only when the second improves, or when a person decides the trade is worth it. Frozen recorded inputs for the other agents (the replay tests from testing) make the local evaluation cheap and repeatable.
In one breath#
At scale and with many agents, new failure modes appear that no single step shows. Speculate only on work without side effects and still pass the guard. Prevent deadlock by structure: requests between agents in one direction, every wait limited by the orchestrator. Catch livelock with budgets across agents, and starvation with a written fairness policy for the queue and for the people. And evaluate the system on whole runs (outcome, process, cost, behavior with a part down) before attributing failures to agents.
What comes next#
Every idea on this site has now had its own page. The last page puts them back together on one case: the insurance lab runs the quote workflow end to end, with its states, its cases, a fixed pipeline next to a bounded agent loop, a crash, a pause for the underwriter, and the code that ran them.