All chapters
  1. Start hereSomething has to decide what runs next
  2. Foundations · 1Sequence, branch and loop: the three primitives of every workflow
  3. Foundations · 2State machines, DAGs, events and agent loops: four shapes of control
  4. Patterns · 3One agent with tools, or several specialized agents
  5. Patterns · 4Supervisor, planner-executor, router, swarm and peers: who talks to whom
  6. Patterns · 5Control plane and data plane: where code decides and where a model may
  7. Mechanics · 6Task queues, fan-out and fan-in, concurrency and rate limits
  8. Mechanics · 7Handoffs: what one agent must give the next
  9. Mechanics · 8Human approvals: pause, wait for a person, resume safely
  10. Production · 9Durable workflows: checkpoints, retries, compensation and cancellation
  11. Production · 10Permissions, budgets and termination: what a run may do, spend, and when it must stop
  12. Production · 11Failure propagation, circuit breakers and fallback routes
  13. Production · 12Trace propagation and outcome evaluation across a workflow
  14. Reference · 13Frameworks: LangGraph, Temporal, AutoGen, CrewAI and the provider SDKs
  15. Reference · 14Testing orchestration: transition coverage and fault injection
  16. Reference · 15Advanced: speculative execution, deadlocks, fairness and evaluating many agents
  17. Reference · 16The insurance lab: one quote workflow, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Production · 11

Failure propagation, circuit breakers and fallback routes

One step fails. Which other steps must stop, which may continue, when to stop calling a broken dependency, and which other route still leads to an acceptable result.

What you will understand
Where a failure stops
Which failures stay inside one step, which end the run, and who draws that line.
Stop calling
How a circuit breaker protects runs from a provider that is down, and what it costs when the provider recovers.
Another way through
Fallback routes and the dead-letter list: a useful result for the underwriter even when a part is broken.

On a Monday morning the roof-data provider starts failing. Not slowly: every call returns an error. The first quote request of the day is UW-908, this site's own case. Its run calls get_roof_assessment, gets an error, retries, gets another error. The fan-in rule says the roof assessment is required, so the run takes the fallback route and recommends request_info with roof_assessment in missing. The underwriter will ask the broker for an inspection report. So far, so good.

Then the second request arrives, and the third, and the tenth. Each run calls the broken provider twice, waits for each error, and ends in the same place. A hundred runs make two hundred calls that everyone already knows will fail. If the provider is overloaded rather than down, those calls are part of why it stays down.

And a quieter question sits under this one. In UW-908 the failure of one lookup did not stop the run. In UW-008, a failure of the guard did stop the run, before the write. Both are failures of one step. Who decided that one may continue and the other may not?

This page follows a failure through the quote workflow: where it is caught, where it travels, when the orchestrator should stop calling a dependency at all, and which other route still gives the underwriter something useful. One note before any output: the agents here are scripted stand-ins and time is simulated, so these results show how the orchestration works, not how good or how fast any model is.

1. Failure propagation: what a failure touches#

When one part fails, other parts are affected: the step that waited for it, the steps after it, the run as a whole, other runs that use the same provider. Failure propagation is the path a failure takes from where it happens to everything it affects. The design question is never "can this fail?" (everything can) but "how far may this failure travel?".

There are two directions. A failure travels up: a lookup fails, so the step gather_evidence must decide whether it failed too, and if it did, the run must decide whether it fails. A failure also travels sideways: the provider fails for one run, and every other run calling it at the same time meets the same failure.

The tool for "up" is an error boundary: a place in the workflow where a failure is caught and turned into an ordinary result, so it travels no further. In the example, every step is an error boundary. A step that raises StepError is caught by the orchestrator in Engine.run_step; the step is marked failed, the attempt is recorded, and if attempts remain it is retried. Nothing outside the step sees an exception.

Inside gather_evidence there is a finer boundary for each of the three lookups. A failed lookup does not fail the step. It becomes a status in the fan-in: ok, failed, skipped. The fan-in rule then decides. Here is UW-903, where similar_decisions fails twice and guidance is slow:

$ $PY run.py --case UW-903
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-903 · One lookup fails and one is slow · dag mode · run-602c · trace 21ca3b4648ab...
  path:      intake > gather_evidence > check_eligibility > draft_recommendation > verify_process > record_recommendation > await_underwriter
  attempts:  find_similar_decisions attempt_number 2 error
  fan-in:    roof_assessment ok [ok], guidance ok [slow], similar_decisions failed [error, error] · missing similar_decisions
  guard:     pass (Eligibility ran and every cited rule was retrieved.)
  recommendation: decline · cites R-ROOF-01 · missing - · policy 2026.2 · stored row status stored
  human_decision: none yet (waiting for: underwriter)
  status:    suspended · model calls 3 · tool calls 5 · simulated time 4.3 s
  expected:  as stated in the case
wrote 1 run file(s) in out/runs/ and out/summary.json

The optional lookup failed, so the failure stopped at the fan-in. The run continued, and the record says what is missing. That is a partial failure: some parts of a step failed and the step still produced a usable result, with the gap stated. The fan-in rule is the policy that decides when partial is good enough. It is the subject of concurrency; this page cares about what happens when it is not.

Plain English. An error boundary is a fire door. The fire is real, but it stays in one room, and the rest of the building gets a clear message about which room is closed.

2. When a failure must end the run#

Some failures must not be contained. Case UW-008 in loop mode: the stand-in model skips search_guidance and drafts a decline that cites R-ROOF-01. The recommendation is even correct. But the guard in step 5, verify_process, checks that every cited rule was retrieved, and this one was not. The guard blocks. The real run:

$ $PY run.py --case UW-008 --mode loop
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-008 · Right recommendation, required check skipped · loop mode · run-e220 · trace e9d1d3c00fc3...
  path:      intake > agent_loop > verify_process
  loop:      get_roof_assessment > find_similar_decisions > check_eligibility > finish · ended: finished · turns 4 · tool calls 3 · tokens 3900
  guard:     block (The draft cites R-ROOF-01, but no guidance search returned it.)
  recommendation: none stored (rows written: 0)
  human_decision: none
  status:    failed · model calls 5 · tool calls 3 · simulated time 4.4 s
  expected:  as stated in the case
wrote 1 run file(s) in out/runs/ and out/summary.json

The run failed. Steps 6, 7 and 8 were marked skipped. Nothing was stored. That is the right propagation, and it was chosen in code: verify_process has one attempt and no fallback route, so its failure goes up to the run. A guard that could be retried until it passes, or skipped when it fails, would not be a guard.

When a run fails, it must not vanish. Engine.fail puts it on a list for people:

def fail(self, run: dict[str, Any], step: str, reason: str) -> dict[str, Any]:
    """End the run as failed. Remaining steps are skipped; the run goes to the dead-letter list for a person."""
    for name in self.wf.order:
        if run["steps"][name]["status"] == "pending":
            self.set_step_status(run, name, "skip")
    self.set_run_status(run, "step_failed")
    self.close_root(run)
    self.event(run, "run_failed", step, {"reason": reason})
    self.services.dead_letter.append({"run_id": run["run_id"], "step": step, "reason": reason})
    self.checkpoint(run, step)
    return run

A dead-letter list (or queue) holds work that could not be completed automatically, with the reason, so that a person can look at it. The broker of UW-008 is still waiting for an answer. A person on the dead-letter list sees run run-e220, step verify_process, and the reason in one sentence, and can start a new run in pipeline mode.

So the line between "contain" and "propagate" is a design decision written into each step: its attempts, its fallback, and whether the run may continue without it. The table states it for the quote workflow.

Step failsContained or propagatedWhy
Optional lookup (similar_decisions)Contained at the fan-in; reported as missingThe rules can run without it (UW-903)
Required lookup (roof_assessment), after its retryContained; fallback route to request_infoA useful recommendation still exists: ask for the fact (UW-006)
check_eligibilityPropagated: run failsNo recommendation without the rules
verify_process blocksPropagated: run fails, nothing storedThe guard exists to stop the write (UW-008)
record_recommendation, after retriesPropagated: run failsThe underwriter cannot see an unstored recommendation
For a backend engineer. A Spring @ControllerAdvice turns exceptions into responses at one boundary; a Kafka consumer sends a poison message to a dead-letter topic after its retries. Both ideas appear here. The difference is the unit: in a workflow the boundary is per step and the dead-letter entry is a whole run with its checkpoint, so a person can resume or restart it, not only read it.

Diagram: How far a failure travels · contained at the fan-in, or up to the run

Read left to right. A failed lookup is caught inside gather_evidence and becomes a status. The fan-in rule decides: an optional lookup is reported missing, a required one sends the run to the fallback route. A guard that blocks is not contained: the run fails before the write and goes to the dead-letter list. The agents are scripted stand-ins and time is simulated.

  1. 1 · Where it fails
    1. A lookup fails: One of the three lookups in step 2 does not return a usable result after its retry.
    2. The guard blocks: Code finds that the draft cites a rule no search returned.
  2. 2 · The boundary
    1. Fan-in rule: The lookup's failure stops here and becomes a status the rule reads.
  3. 3 · The route
    1. Continue: An optional lookup is missing; the run continues and says so.
    2. Fallback route: A required lookup is missing; the recommendation asks for the fact instead of guessing it.
  4. 4 · Up to the run
    1. Run fails: The guard's failure is not contained by design. Nothing is stored.
    2. Dead-letter list: The failed run is kept with its step and reason so a person can act.

3. Timeouts: the failure that does not announce itself#

An error arrives quickly and says what went wrong. A hang says nothing. A timeout is a limit on how long the orchestrator waits for one call or one step before it treats the call as failed. Without one, a single slow provider holds a worker, then the next one, until no worker is left for anything else.

In the example the roof lookup of UW-006 times out after 2,000 simulated milliseconds per attempt (an illustrative value in core.py:OUTCOME_MS). Two attempts, a 500 ms backoff between them, and the fan-in has its answer at 5,900 ms. The approval wait has its own timeout, 24 simulated hours, with a different reaction: escalate, not fail (see human approvals).

Common confusion. A timeout does not mean the call failed on the other side. It means the orchestrator stopped waiting. The provider may still finish the work. That is harmless for a lookup, and the reason the write in step 6 carries an idempotency key (see durable workflows).

4. The circuit breaker: stop calling what is broken#

Back to Monday morning. Each run contains its own failure well. The problem is that each run pays for it separately: two calls, two waits, every time. The runs need to share what they learned.

A circuit breaker is a small state machine in front of a dependency, shared by all runs, that stops calls after repeated failures and lets one call through later to test whether the dependency has recovered. It has three states:

The logic is two pure functions in the example. The first runs before a call:

def breaker_before_call(breaker: dict[str, Any], cooldown_runs: int) -> dict[str, Any]:
    """May this run call the provider? Open: no, until `cooldown_runs` runs have passed; then half-open: one try."""
    b = dict(breaker)
    if b["state"] != "open":
        return {"breaker": b, "allowed": True}
    b["open_runs"] += 1
    if b["open_runs"] > cooldown_runs:
        b["state"] = "half_open"
        return {"breaker": b, "allowed": True}
    return {"breaker": b, "allowed": False}

The second, breaker_after_call, records the result: success resets to closed, a failure in half_open reopens it, and the threshold-th failure in a row opens it. The cooldown is counted in runs here, because the example's clock is simulated per run; a production breaker counts time.

UW-908 plays ten runs against one shared breaker: five while the provider is down, five after it recovers. The real output:

$ $PY run.py --case UW-908
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-908 · The roof provider fails for several runs in a row · dag mode · run-2412 · trace e879f36b8f1e...
  breaker sequence (the runs share one breaker; this run is the first that meets it open):
    run  1 · provider would fail · breaker at call closed    after closed · provider calls 2 · route request_info
    run  2 · provider would fail · breaker at call closed    after closed · provider calls 2 · route request_info
    run  3 · provider would fail · breaker at call closed    after open   · provider calls 2 · route request_info
    run  4 · provider would fail · breaker at call open      after open   · provider calls 0 · route request_info
    run  5 · provider would fail · breaker at call open      after open   · provider calls 0 · route request_info
    run  6 · provider would ok   · breaker at call half_open after closed · provider calls 1 · route approve
    run  7 · provider would ok   · breaker at call closed    after closed · provider calls 1 · route approve
    run  8 · provider would ok   · breaker at call closed    after closed · provider calls 1 · route approve
    run  9 · provider would ok   · breaker at call closed    after closed · provider calls 1 · route approve
    run 10 · provider would ok   · breaker at call closed    after closed · provider calls 1 · route approve
  path:      intake > gather_evidence > check_eligibility > draft_recommendation > verify_process > record_recommendation > await_underwriter
  fan-in:    roof_assessment skipped [not called], guidance ok [ok], similar_decisions ok [ok] · missing roof_assessment
  guard:     pass (Eligibility ran and every cited rule was retrieved.)
  recommendation: request_info · cites - · missing roof_assessment · policy 2026.2 · stored row status stored
  human_decision: none yet (waiting for: underwriter)
  status:    suspended · model calls 3 · tool calls 3 · simulated time 3.1 s
  expected:  as stated in the case
wrote 1 run file(s) in out/runs/ and out/summary.json

Runs 4 and 5 did not call the provider at all and reached the same route as runs 1 to 3, faster. Run 6 was the test call, it succeeded, and the breaker closed. The detail printed below the sequence is run 4, the first run that met the breaker open: its roof lookup is skipped, not failed, and the run still produced a useful recommendation.

The threshold is a trade, not a constant. The script below runs the same ten provider outcomes with two thresholds. It ran in a scratch copy of the example.

import core

provider = ["error"] * 5 + ["ok"] * 5          # ten runs in a row, illustrative
for threshold in (3, 5):
    rows = core.simulate_breaker(provider, threshold=threshold, cooldown_runs=2)
    at_call = " ".join(f"{r['state_at_call']:9}" for r in rows)
    called = sum(r["called"] for r in rows)
    print(f"threshold {threshold}: provider called in {called} of 10 runs")
    print(f"  at call: {at_call}")
threshold 3: provider called in 8 of 10 runs
  at call: closed    closed    closed    open      open      half_open closed    closed    closed    closed   
threshold 5: provider called in 8 of 10 runs
  at call: closed    closed    closed    closed    closed    open      open      half_open closed    closed

Diagram: The circuit breaker's three states · shared by every run

Read left to right. Closed: calls go through and failures in a row are counted. At the threshold the breaker opens and runs take the fallback without calling. After the cooldown one run tests the provider (half-open): success closes it, failure opens it again. Threshold 3 and cooldown 2 runs are illustrative.

  1. Closed
    1. Calls go through: The normal state. A success resets the count.
  2. Open
    1. No calls: After 3 failures in a row runs do not call the provider. The lookup is skipped, not failed.
  3. Half-open
    1. One test call: One run may try. Success closes the breaker; failure opens it again.

Both settings call the provider in 8 of 10 runs, but not the same 8. With threshold 3 the breaker skips two runs during the outage, where the calls were useless. With threshold 5 it never protects anyone during the outage, opens just as the provider recovers, and makes runs 6 and 7 take the fallback route although the provider would have answered. A breaker that opens late costs both ways.

The same idea in a coding agent. A coding agent's CI runs a flaky integration test against a staging service that is down. Rerunning the whole pipeline after every edit spends the agent's turns on a failure the agent cannot fix. A shared signal (the service's status, or a breaker on that test job) lets the agent stop, report the outside failure, and wait. CI and its gates are covered on Diff to Deploy.

5. Fallback routes#

A fallback route is a different path through the workflow, chosen in code when the normal path cannot be completed, that still ends in an acceptable result. The quote workflow has one main fallback: when the required roof assessment is missing (after retries, or because the breaker is open), the rules step records request_info, and the underwriter asks the broker for the roof age or an inspection.

Why is that acceptable? Because the outcome is honest. It does not guess the roof age, and it does not pretend the evidence is complete. R-ROOF-02 in the policy fixture says the same thing for a roof age that is simply unknown (UW-003). A fallback that invents a value to keep the normal path going is not a fallback; it is a silent wrong answer.

What would not be an acceptable fallback here: asking a model to estimate the roof age from the building's year of construction, or using the age from the application alone when the whole point of the lookup is to check it (UW-004 shows why). A fallback must be designed as carefully as the main path, and tested on purpose: the testing page injects each failure and checks the route.

What this is not. This page decides where a failure goes inside one workflow. Operational guarantees around it (service levels, load shedding, incident response, and how retries behave under load across a whole system) are the subject of Agent Reliability Engineering.

6. Try it: break the provider#

Two labs. The first sets the roof provider's result for each of ten runs, and the breaker's threshold and cooldown. It shows closed, open and half-open per run, which runs called the provider, and which took the fallback route.

RunProviderBreaker at call (threshold 3, cooldown 2)CalledRoute
1 to 3errorclosedyes, twice each (retry)fallback: request_info
4 and 5erroropennofallback: request_info
6okhalf-openyes, oncenormal: approve
7 to 10okclosedyesnormal: approve

The second lab is the fan-out of step 2. Set each lookup to ok, slow, timeout or error, and see which failures stay inside the fan-in and which change the route.

LookupsFan-in resultRoute
all okall three aggregatednormal
similar_decisions error, guidance slow (UW-903)optional lookup missing, reportednormal, with the gap stated
roof_assessment timeout twice (UW-006)required lookup missingfallback: request_info
roof_assessment skipped by an open breaker (UW-908)required lookup missingfallback: request_info, without waiting

Who decides here#

DecisionWho
Whether a failure stays inside a step or ends the runCode: each step's attempts, fan-in rule and fallback, written at design time
How long to wait for a callCode: a timeout per call
Whether a run may call the provider at allCode: the shared circuit breaker
Which route a run takes when a required fact is missingCode: the fan-in rule, then the rules step
What happens to a failed runA person, from the dead-letter list
What a model decides hereNothing. A model's output can be the thing that fails a guard; it never decides where the failure goes

Checklist#

Tuning a breaker, and breakers per whatThresholds, error rates, scope and the half-open stampede

The example counts failures in a row. Production breakers often use an error rate over a window instead (for example, more than half of the calls in the last minute), because a provider that fails every other call never produces three failures in a row and still deserves protection. The window must hold enough calls for the rate to mean something; at low traffic a rate breaker either never opens or opens on two calls.

Scope matters as much as threshold. One breaker per provider is the default. A breaker per provider and region, or per operation, avoids opening on calls that would succeed. A breaker shared across many orchestrator processes needs shared state, or each process learns the outage separately.

Half-open needs care under load. If every process lets "one" test call through at the same moment, the recovering provider gets a burst. Production breakers limit half-open to a small number of calls overall, and some add jitter to the cooldown.

Retries multiplyNested retries and the arithmetic of an outage

Retries at several levels multiply. If the HTTP client retries 3 times, the lookup retries 2 times and the step retries 2 times, one logical call can become 12 requests against a provider that is already failing. The example retries at one level only: the lookup inside the fan-out, max_attempts 2. Decide at which level a retry lives, and make every other level fail fast.

Backoff spreads retries in time; the example uses a fixed 500 ms. Exponential backoff with jitter is the usual production choice, so that many runs that failed together do not retry together. A breaker and a retry budget (a cap on the share of calls that may be retries) are what keep an outage from becoming a load test. Operational detail is the subject of Agent Reliability Engineering.

When the model is the failing partMalformed output, refusals, and the fallback that is a person

A model call can fail like a provider (an error, a timeout, an overloaded endpoint) and the same tools apply: timeout, retry with a limit, breaker, fallback. It can also fail in ways that return successfully: output that does not parse, a draft that cites a rule it never retrieved (UW-008), reasons that disagree with the rule outcome (UW-907). Those failures are caught by checks on the output, in code, at the exit of the step. Which checks, and how output is validated and repaired, belong to Structured Output Engineering and Guardrails Engineering.

The usual fallback for a model that cannot produce a checked draft is not another model. In the quote workflow it is the senior underwriter: UW-907's guard routes to escalate. A second model as fallback is possible, but it needs the same guard at its exit and its own place in the evaluation, or the fallback becomes the path nobody measured.

In one breath#

Every step is an error boundary, and each step's design says whether its failure stays inside it or ends the run: optional lookups are reported missing, a required lookup takes the fallback route to request_info, and a blocked guard fails the run before the write and sends it to the dead-letter list. Every call has a timeout. A circuit breaker shared by all runs stops calling a provider that keeps failing, and lets one run test it later. A fallback states what is missing; it never invents it.

What comes next#

You can now decide where every failure goes. To see that it went there, you need a record of the run that crosses every boundary on this page: the retries, the skipped call, the fallback, the guard, the dead-letter entry, and later the pause for a person. The next page, tracing, keeps one trace_id across agents, queues and a human pause, and asks how to judge the outcome of a whole run instead of one step.