On a Monday morning the roof-data provider starts failing. Not slowly: every call returns an error. The first quote request of the day is UW-908, this site's own case. Its run calls get_roof_assessment, gets an error, retries, gets another error. The fan-in rule says the roof assessment is required, so the run takes the fallback route and recommends request_info with roof_assessment in missing. The underwriter will ask the broker for an inspection report. So far, so good.
Then the second request arrives, and the third, and the tenth. Each run calls the broken provider twice, waits for each error, and ends in the same place. A hundred runs make two hundred calls that everyone already knows will fail. If the provider is overloaded rather than down, those calls are part of why it stays down.
And a quieter question sits under this one. In UW-908 the failure of one lookup did not stop the run. In UW-008, a failure of the guard did stop the run, before the write. Both are failures of one step. Who decided that one may continue and the other may not?
This page follows a failure through the quote workflow: where it is caught, where it travels, when the orchestrator should stop calling a dependency at all, and which other route still gives the underwriter something useful. One note before any output: the agents here are scripted stand-ins and time is simulated, so these results show how the orchestration works, not how good or how fast any model is.
1. Failure propagation: what a failure touches#
When one part fails, other parts are affected: the step that waited for it, the steps after it, the run as a whole, other runs that use the same provider. Failure propagation is the path a failure takes from where it happens to everything it affects. The design question is never "can this fail?" (everything can) but "how far may this failure travel?".
There are two directions. A failure travels up: a lookup fails, so the step gather_evidence must decide whether it failed too, and if it did, the run must decide whether it fails. A failure also travels sideways: the provider fails for one run, and every other run calling it at the same time meets the same failure.
The tool for "up" is an error boundary: a place in the workflow where a failure is caught and turned into an ordinary result, so it travels no further. In the example, every step is an error boundary. A step that raises StepError is caught by the orchestrator in Engine.run_step; the step is marked failed, the attempt is recorded, and if attempts remain it is retried. Nothing outside the step sees an exception.
Inside gather_evidence there is a finer boundary for each of the three lookups. A failed lookup does not fail the step. It becomes a status in the fan-in: ok, failed, skipped. The fan-in rule then decides. Here is UW-903, where similar_decisions fails twice and guidance is slow:
$ $PY run.py --case UW-903
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-903 · One lookup fails and one is slow · dag mode · run-602c · trace 21ca3b4648ab...
path: intake > gather_evidence > check_eligibility > draft_recommendation > verify_process > record_recommendation > await_underwriter
attempts: find_similar_decisions attempt_number 2 error
fan-in: roof_assessment ok [ok], guidance ok [slow], similar_decisions failed [error, error] · missing similar_decisions
guard: pass (Eligibility ran and every cited rule was retrieved.)
recommendation: decline · cites R-ROOF-01 · missing - · policy 2026.2 · stored row status stored
human_decision: none yet (waiting for: underwriter)
status: suspended · model calls 3 · tool calls 5 · simulated time 4.3 s
expected: as stated in the case
wrote 1 run file(s) in out/runs/ and out/summary.json
The optional lookup failed, so the failure stopped at the fan-in. The run continued, and the record says what is missing. That is a partial failure: some parts of a step failed and the step still produced a usable result, with the gap stated. The fan-in rule is the policy that decides when partial is good enough. It is the subject of concurrency; this page cares about what happens when it is not.
Plain English. An error boundary is a fire door. The fire is real, but it stays in one room, and the rest of the building gets a clear message about which room is closed.
2. When a failure must end the run#
Some failures must not be contained. Case UW-008 in loop mode: the stand-in model skips search_guidance and drafts a decline that cites R-ROOF-01. The recommendation is even correct. But the guard in step 5, verify_process, checks that every cited rule was retrieved, and this one was not. The guard blocks. The real run:
$ $PY run.py --case UW-008 --mode loop
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-008 · Right recommendation, required check skipped · loop mode · run-e220 · trace e9d1d3c00fc3...
path: intake > agent_loop > verify_process
loop: get_roof_assessment > find_similar_decisions > check_eligibility > finish · ended: finished · turns 4 · tool calls 3 · tokens 3900
guard: block (The draft cites R-ROOF-01, but no guidance search returned it.)
recommendation: none stored (rows written: 0)
human_decision: none
status: failed · model calls 5 · tool calls 3 · simulated time 4.4 s
expected: as stated in the case
wrote 1 run file(s) in out/runs/ and out/summary.json
The run failed. Steps 6, 7 and 8 were marked skipped. Nothing was stored. That is the right propagation, and it was chosen in code: verify_process has one attempt and no fallback route, so its failure goes up to the run. A guard that could be retried until it passes, or skipped when it fails, would not be a guard.
When a run fails, it must not vanish. Engine.fail puts it on a list for people:
def fail(self, run: dict[str, Any], step: str, reason: str) -> dict[str, Any]:
"""End the run as failed. Remaining steps are skipped; the run goes to the dead-letter list for a person."""
for name in self.wf.order:
if run["steps"][name]["status"] == "pending":
self.set_step_status(run, name, "skip")
self.set_run_status(run, "step_failed")
self.close_root(run)
self.event(run, "run_failed", step, {"reason": reason})
self.services.dead_letter.append({"run_id": run["run_id"], "step": step, "reason": reason})
self.checkpoint(run, step)
return run
A dead-letter list (or queue) holds work that could not be completed automatically, with the reason, so that a person can look at it. The broker of UW-008 is still waiting for an answer. A person on the dead-letter list sees run run-e220, step verify_process, and the reason in one sentence, and can start a new run in pipeline mode.
So the line between "contain" and "propagate" is a design decision written into each step: its attempts, its fallback, and whether the run may continue without it. The table states it for the quote workflow.
| Step fails | Contained or propagated | Why |
|---|---|---|
Optional lookup (similar_decisions) | Contained at the fan-in; reported as missing | The rules can run without it (UW-903) |
Required lookup (roof_assessment), after its retry | Contained; fallback route to request_info | A useful recommendation still exists: ask for the fact (UW-006) |
check_eligibility | Propagated: run fails | No recommendation without the rules |
verify_process blocks | Propagated: run fails, nothing stored | The guard exists to stop the write (UW-008) |
record_recommendation, after retries | Propagated: run fails | The underwriter cannot see an unstored recommendation |
For a backend engineer. A Spring @ControllerAdvice turns exceptions into responses at one boundary; a Kafka consumer sends a poison message to a dead-letter topic after its retries. Both ideas appear here. The difference is the unit: in a workflow the boundary is per step and the dead-letter entry is a whole run with its checkpoint, so a person can resume or restart it, not only read it.
Diagram: How far a failure travels · contained at the fan-in, or up to the run
Read left to right. A failed lookup is caught inside gather_evidence and becomes a status. The fan-in rule decides: an optional lookup is reported missing, a required one sends the run to the fallback route. A guard that blocks is not contained: the run fails before the write and goes to the dead-letter list. The agents are scripted stand-ins and time is simulated.
- 1 · Where it fails
- A lookup fails: One of the three lookups in step 2 does not return a usable result after its retry.
- The guard blocks: Code finds that the draft cites a rule no search returned.
- 2 · The boundary
- Fan-in rule: The lookup's failure stops here and becomes a status the rule reads.
- 3 · The route
- Continue: An optional lookup is missing; the run continues and says so.
- Fallback route: A required lookup is missing; the recommendation asks for the fact instead of guessing it.
- 4 · Up to the run
- Run fails: The guard's failure is not contained by design. Nothing is stored.
- Dead-letter list: The failed run is kept with its step and reason so a person can act.
3. Timeouts: the failure that does not announce itself#
An error arrives quickly and says what went wrong. A hang says nothing. A timeout is a limit on how long the orchestrator waits for one call or one step before it treats the call as failed. Without one, a single slow provider holds a worker, then the next one, until no worker is left for anything else.
In the example the roof lookup of UW-006 times out after 2,000 simulated milliseconds per attempt (an illustrative value in core.py:OUTCOME_MS). Two attempts, a 500 ms backoff between them, and the fan-in has its answer at 5,900 ms. The approval wait has its own timeout, 24 simulated hours, with a different reaction: escalate, not fail (see human approvals).
Common confusion. A timeout does not mean the call failed on the other side. It means the orchestrator stopped waiting. The provider may still finish the work. That is harmless for a lookup, and the reason the write in step 6 carries an idempotency key (see durable workflows).
4. The circuit breaker: stop calling what is broken#
Back to Monday morning. Each run contains its own failure well. The problem is that each run pays for it separately: two calls, two waits, every time. The runs need to share what they learned.
A circuit breaker is a small state machine in front of a dependency, shared by all runs, that stops calls after repeated failures and lets one call through later to test whether the dependency has recovered. It has three states:
- closed: calls go through; failures are counted.
- open: after a number of failures in a row (the threshold), calls are not made at all. The run takes its fallback route at once.
- half-open: after a cooldown, one run is allowed to try. Success closes the breaker; failure opens it again.
The logic is two pure functions in the example. The first runs before a call:
def breaker_before_call(breaker: dict[str, Any], cooldown_runs: int) -> dict[str, Any]:
"""May this run call the provider? Open: no, until `cooldown_runs` runs have passed; then half-open: one try."""
b = dict(breaker)
if b["state"] != "open":
return {"breaker": b, "allowed": True}
b["open_runs"] += 1
if b["open_runs"] > cooldown_runs:
b["state"] = "half_open"
return {"breaker": b, "allowed": True}
return {"breaker": b, "allowed": False}
The second, breaker_after_call, records the result: success resets to closed, a failure in half_open reopens it, and the threshold-th failure in a row opens it. The cooldown is counted in runs here, because the example's clock is simulated per run; a production breaker counts time.
UW-908 plays ten runs against one shared breaker: five while the provider is down, five after it recovers. The real output:
$ $PY run.py --case UW-908
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-908 · The roof provider fails for several runs in a row · dag mode · run-2412 · trace e879f36b8f1e...
breaker sequence (the runs share one breaker; this run is the first that meets it open):
run 1 · provider would fail · breaker at call closed after closed · provider calls 2 · route request_info
run 2 · provider would fail · breaker at call closed after closed · provider calls 2 · route request_info
run 3 · provider would fail · breaker at call closed after open · provider calls 2 · route request_info
run 4 · provider would fail · breaker at call open after open · provider calls 0 · route request_info
run 5 · provider would fail · breaker at call open after open · provider calls 0 · route request_info
run 6 · provider would ok · breaker at call half_open after closed · provider calls 1 · route approve
run 7 · provider would ok · breaker at call closed after closed · provider calls 1 · route approve
run 8 · provider would ok · breaker at call closed after closed · provider calls 1 · route approve
run 9 · provider would ok · breaker at call closed after closed · provider calls 1 · route approve
run 10 · provider would ok · breaker at call closed after closed · provider calls 1 · route approve
path: intake > gather_evidence > check_eligibility > draft_recommendation > verify_process > record_recommendation > await_underwriter
fan-in: roof_assessment skipped [not called], guidance ok [ok], similar_decisions ok [ok] · missing roof_assessment
guard: pass (Eligibility ran and every cited rule was retrieved.)
recommendation: request_info · cites - · missing roof_assessment · policy 2026.2 · stored row status stored
human_decision: none yet (waiting for: underwriter)
status: suspended · model calls 3 · tool calls 3 · simulated time 3.1 s
expected: as stated in the case
wrote 1 run file(s) in out/runs/ and out/summary.json
Runs 4 and 5 did not call the provider at all and reached the same route as runs 1 to 3, faster. Run 6 was the test call, it succeeded, and the breaker closed. The detail printed below the sequence is run 4, the first run that met the breaker open: its roof lookup is skipped, not failed, and the run still produced a useful recommendation.
The threshold is a trade, not a constant. The script below runs the same ten provider outcomes with two thresholds. It ran in a scratch copy of the example.
import core
provider = ["error"] * 5 + ["ok"] * 5 # ten runs in a row, illustrative
for threshold in (3, 5):
rows = core.simulate_breaker(provider, threshold=threshold, cooldown_runs=2)
at_call = " ".join(f"{r['state_at_call']:9}" for r in rows)
called = sum(r["called"] for r in rows)
print(f"threshold {threshold}: provider called in {called} of 10 runs")
print(f" at call: {at_call}")
threshold 3: provider called in 8 of 10 runs
at call: closed closed closed open open half_open closed closed closed closed
threshold 5: provider called in 8 of 10 runs
at call: closed closed closed closed closed open open half_open closed closed
Diagram: The circuit breaker's three states · shared by every run
Read left to right. Closed: calls go through and failures in a row are counted. At the threshold the breaker opens and runs take the fallback without calling. After the cooldown one run tests the provider (half-open): success closes it, failure opens it again. Threshold 3 and cooldown 2 runs are illustrative.
- Closed
- Calls go through: The normal state. A success resets the count.
- Open
- No calls: After 3 failures in a row runs do not call the provider. The lookup is skipped, not failed.
- Half-open
- One test call: One run may try. Success closes the breaker; failure opens it again.
Both settings call the provider in 8 of 10 runs, but not the same 8. With threshold 3 the breaker skips two runs during the outage, where the calls were useless. With threshold 5 it never protects anyone during the outage, opens just as the provider recovers, and makes runs 6 and 7 take the fallback route although the provider would have answered. A breaker that opens late costs both ways.
The same idea in a coding agent. A coding agent's CI runs a flaky integration test against a staging service that is down. Rerunning the whole pipeline after every edit spends the agent's turns on a failure the agent cannot fix. A shared signal (the service's status, or a breaker on that test job) lets the agent stop, report the outside failure, and wait. CI and its gates are covered on Diff to Deploy.
5. Fallback routes#
A fallback route is a different path through the workflow, chosen in code when the normal path cannot be completed, that still ends in an acceptable result. The quote workflow has one main fallback: when the required roof assessment is missing (after retries, or because the breaker is open), the rules step records request_info, and the underwriter asks the broker for the roof age or an inspection.
Why is that acceptable? Because the outcome is honest. It does not guess the roof age, and it does not pretend the evidence is complete. R-ROOF-02 in the policy fixture says the same thing for a roof age that is simply unknown (UW-003). A fallback that invents a value to keep the normal path going is not a fallback; it is a silent wrong answer.
What would not be an acceptable fallback here: asking a model to estimate the roof age from the building's year of construction, or using the age from the application alone when the whole point of the lookup is to check it (UW-004 shows why). A fallback must be designed as carefully as the main path, and tested on purpose: the testing page injects each failure and checks the route.
What this is not. This page decides where a failure goes inside one workflow. Operational guarantees around it (service levels, load shedding, incident response, and how retries behave under load across a whole system) are the subject of Agent Reliability Engineering.
6. Try it: break the provider#
Two labs. The first sets the roof provider's result for each of ten runs, and the breaker's threshold and cooldown. It shows closed, open and half-open per run, which runs called the provider, and which took the fallback route.
| Run | Provider | Breaker at call (threshold 3, cooldown 2) | Called | Route |
|---|---|---|---|---|
| 1 to 3 | error | closed | yes, twice each (retry) | fallback: request_info |
| 4 and 5 | error | open | no | fallback: request_info |
| 6 | ok | half-open | yes, once | normal: approve |
| 7 to 10 | ok | closed | yes | normal: approve |
The second lab is the fan-out of step 2. Set each lookup to ok, slow, timeout or error, and see which failures stay inside the fan-in and which change the route.
| Lookups | Fan-in result | Route |
|---|---|---|
| all ok | all three aggregated | normal |
similar_decisions error, guidance slow (UW-903) | optional lookup missing, reported | normal, with the gap stated |
roof_assessment timeout twice (UW-006) | required lookup missing | fallback: request_info |
roof_assessment skipped by an open breaker (UW-908) | required lookup missing | fallback: request_info, without waiting |
Who decides here#
| Decision | Who |
|---|---|
| Whether a failure stays inside a step or ends the run | Code: each step's attempts, fan-in rule and fallback, written at design time |
| How long to wait for a call | Code: a timeout per call |
| Whether a run may call the provider at all | Code: the shared circuit breaker |
| Which route a run takes when a required fact is missing | Code: the fan-in rule, then the rules step |
| What happens to a failed run | A person, from the dead-letter list |
| What a model decides here | Nothing. A model's output can be the thing that fails a guard; it never decides where the failure goes |
Checklist#
- Every step has a written answer to "if this fails after its retries, does the run continue?".
- Every call has a timeout, and the timeout is shorter than the deadline of whatever waits for it.
- A guard has one attempt and no fallback: when it blocks, nothing is written.
- A failed run goes to a dead-letter list with its step, its reason and its checkpoint.
- A shared dependency has a circuit breaker whose threshold and cooldown are chosen from the provider's failure pattern, and logged.
- A fallback route never invents the missing fact; it states what is missing.
- Each failure is injected in a test, and the route is checked.
Tuning a breaker, and breakers per whatThresholds, error rates, scope and the half-open stampede
The example counts failures in a row. Production breakers often use an error rate over a window instead (for example, more than half of the calls in the last minute), because a provider that fails every other call never produces three failures in a row and still deserves protection. The window must hold enough calls for the rate to mean something; at low traffic a rate breaker either never opens or opens on two calls.
Scope matters as much as threshold. One breaker per provider is the default. A breaker per provider and region, or per operation, avoids opening on calls that would succeed. A breaker shared across many orchestrator processes needs shared state, or each process learns the outage separately.
Half-open needs care under load. If every process lets "one" test call through at the same moment, the recovering provider gets a burst. Production breakers limit half-open to a small number of calls overall, and some add jitter to the cooldown.
Retries multiplyNested retries and the arithmetic of an outage
Retries at several levels multiply. If the HTTP client retries 3 times, the lookup retries 2 times and the step retries 2 times, one logical call can become 12 requests against a provider that is already failing. The example retries at one level only: the lookup inside the fan-out, max_attempts 2. Decide at which level a retry lives, and make every other level fail fast.
Backoff spreads retries in time; the example uses a fixed 500 ms. Exponential backoff with jitter is the usual production choice, so that many runs that failed together do not retry together. A breaker and a retry budget (a cap on the share of calls that may be retries) are what keep an outage from becoming a load test. Operational detail is the subject of Agent Reliability Engineering.
When the model is the failing partMalformed output, refusals, and the fallback that is a person
A model call can fail like a provider (an error, a timeout, an overloaded endpoint) and the same tools apply: timeout, retry with a limit, breaker, fallback. It can also fail in ways that return successfully: output that does not parse, a draft that cites a rule it never retrieved (UW-008), reasons that disagree with the rule outcome (UW-907). Those failures are caught by checks on the output, in code, at the exit of the step. Which checks, and how output is validated and repaired, belong to Structured Output Engineering and Guardrails Engineering.
The usual fallback for a model that cannot produce a checked draft is not another model. In the quote workflow it is the senior underwriter: UW-907's guard routes to escalate. A second model as fallback is possible, but it needs the same guard at its exit and its own place in the evaluation, or the fallback becomes the path nobody measured.
In one breath#
Every step is an error boundary, and each step's design says whether its failure stays inside it or ends the run: optional lookups are reported missing, a required lookup takes the fallback route to request_info, and a blocked guard fails the run before the write and sends it to the dead-letter list. Every call has a timeout. A circuit breaker shared by all runs stops calling a provider that keeps failing, and lets one run test it later. A fallback states what is missing; it never invents it.
What comes next#
You can now decide where every failure goes. To see that it went there, you need a record of the run that crosses every boundary on this page: the retries, the skipped call, the fallback, the guard, the dead-letter entry, and later the pause for a person. The next page, tracing, keeps one trace_id across agents, queues and a human pause, and asks how to judge the outcome of a whole run instead of one step.