The quote workflow has eighteen cases. All of them pass. Every run ends with the line expected: as stated in the case. The team is ready to ship a change to the approval logic.
Then someone asks a question nobody can answer from the green list: has any test ever sent a decision from the wrong role? Has a waiting run ever been cancelled in a test? Has the orchestrator ever restarted while a run was suspended? The eighteen cases were written to teach the insurance situations: a disputed roof age, a timeout, an injected passage. They were not written to exercise the orchestrator. So nobody knows whether the code paths that handle those events work, because nothing has ever taken them.
This is the testing problem of orchestration. The cases test the business. The orchestrator is a different program, with its own states and transitions, and most of its bugs hide in the paths that ordinary cases never reach: the second click, the restart during a wait, the cancel that arrives at the wrong moment.
This page tests the orchestrator as what it is: ordinary code. One note before any output: the agents here are scripted stand-ins and time is simulated, so these results show how the orchestration works, not how good or how fast any model is.
1. The orchestrator can be tested without a model#
The first thing to notice is what makes this testable at all. Every decision about what runs next is in code: the transitions, the guard, the fan-in rule, the budget check, the resume check. The parts that are not code (the model's draft, the provider's answer, the underwriter's click) enter the orchestrator as inputs.
So the test replaces those inputs with fixed ones. A stand-in agent is a scripted replacement for a model-backed agent: it returns a fixed or seeded output for each input, without calling a model. The example's agents are all stand-ins (agents_sim.py), its tools return fixed answers per case (tools_sim.py), and its clock is simulated. That is why 45 unit tests run in a fraction of a second and give the same result on every machine.
$ /opt/homebrew/bin/python3 -m unittest
.............................................
----------------------------------------------------------------------
Ran 45 tests in 0.173s
OK
What a stand-in proves: given these outputs from the model and the tools, the orchestrator does the right thing. What it does not prove: that a real model produces such outputs. That second question is measured with many trials and graders, which is the subject of Eval Engineering. Both are needed. A workflow whose orchestrator is untested fails even when the model is right.
For a backend engineer. This is the familiar split between unit tests with mocks and integration tests. A stand-in agent is a mock of the model call, at the level of the agent's output. The comparison stops at variability: a mock usually returns one answer, while a useful stand-in in loop mode returns seeded, varied choices (as in UW-010) so that the tests also cover paths a real model might take.
2. Transition coverage#
Line coverage tells you which lines ran. For an orchestrator the more useful question is which transitions ran. Transition coverage counts, for every legal transition of the run's state machine (from one state, on one event, to another state), whether at least one test took it.
The run state machine of the example has 14 legal transitions, listed in core.py:RUN_TRANSITIONS with a reason each (the shapes page explains state machines). The script below runs the 18 cases in their own modes and counts which transitions they took. It ran in a scratch copy of the example.
import core
from run import load_cases, case_mode, run_case
coverage = {f"{t['from']}:{t['event']}": 0 for t in core.RUN_TRANSITIONS}
for case in load_cases():
for t in run_case(case, case_mode(case))["transitions"]:
coverage[t] += 1
covered = [k for k, v in coverage.items() if v]
print(f"18 cases exercise {len(covered)} of {len(coverage)} run transitions")
for k, v in coverage.items():
if v == 0:
print(" never taken:", k)
18 cases exercise 10 of 14 run transitions
never taken: suspended:signal_rejected
never taken: suspended:approval_timeout
never taken: suspended:process_restarted
never taken: suspended:cancel
All four gaps start from suspended. That is not a coincidence. The cases describe what happens to a submission; what happens to a run while nobody is looking at it (a wrong click, a timeout, a restart, a cancel) is the orchestrator's business, and the cases never asked. Those are exactly the paths the opening question was about.
Diagram: Four ways to test the orchestrator · none of them calls a model
Read left to right. Stand-ins replace the model, the tools and the clock. The 18 cases take 10 of the 14 run transitions; fault runs take the other four, all on the wait for a person. Replay tests compare a recorded run with a fresh one. Invariants are checked over every case and mode.
- 1 · Replace
- Stand-ins: Scripted or seeded outputs instead of model calls; fixed tool answers per case; simulated time.
- 2 · Cover
- 18 cases: The business cases leave every unusual event on the wait untaken.
- Fault runs: Wrong role, restart then silence, cancel while waiting, caused on purpose.
- 3 · Compare
- Replay tests: The same inputs on new code must give the same run, or a person reviews the difference.
- 4 · Always
- Invariants: Short absolute statements checked over all runs.
3. Fault injection#
The four missing transitions will not happen by themselves in a test. They have to be caused. Fault injection means making a failure or an unusual event happen on purpose, at a chosen point, to check how the orchestrator reacts.
The example takes faults as inputs, so injecting one is a function argument. The script below injects three: a decision from an underwriter where a senior underwriter is required, a roof lookup that errors instead of timing out, and a guidance search that times out instead of the roof lookup.
from run import load_cases, run_case
def last(r):
return r["transitions"][-1]
uw004, uw006 = load_cases("UW-004")[0], load_cases("UW-006")[0]
wrong_role = run_case(uw004, signals=[{"decided_by": "underwriter", "decision": "approve", "reason": "x"}])
print(f"UW-004, an underwriter answers an escalation -> {last(wrong_role)}, status {wrong_role['status']}")
roof_down = run_case(uw006, outcome_overrides={"roof_assessment": ["error"]})
print(f"UW-006, roof lookup errors instead of timing out -> route {roof_down['route']}, missing {roof_down['missing']}")
guidance_down = run_case(uw006, outcome_overrides={"roof_assessment": ["ok"], "guidance": ["timeout"]})
print(f"UW-006, guidance times out instead -> route {guidance_down['route']}, missing {guidance_down['missing']}")
UW-004, an underwriter answers an escalation -> suspended:signal_rejected, status suspended
UW-006, roof lookup errors instead of timing out -> route request_info, missing ['roof_assessment']
UW-006, guidance times out instead -> route request_info, missing ['guidance']
The first line takes one of the four missing transitions: the signal is rejected and the run keeps waiting for the right role. The other two check that the fallback route does not depend on how a required lookup fails, and that it names the right missing branch.
The example's data builder (scripts/build_data.py:fault_runs) adds three such runs to the coverage count: the wrong role on UW-004, a restart during the wait followed by 24 hours of silence on UW-002, and a cancel while UW-001 waits. With them, every one of the 14 transitions is taken at least once. The builder refuses to produce the site's data if any transition is left uncovered, so a new transition added to the code without a test breaks the build.
A useful fault catalog for an orchestrator lists, for each step: fails once, fails on every attempt, times out, returns late, returns twice; and for each wait: wrong token, wrong role, duplicate, timeout, restart during the wait, cancel during the wait, the world changes during the wait. Each row is one test.
The same idea in a coding agent. A coding agent's workflow has its own rare paths: CI that fails after the agent has already marked the task done, a reviewer who requests changes on a pull request the agent believes is finished, a branch that was force-pushed underneath it. Each can be injected in a test harness with a fake CI result or a fake review event, long before it happens on a real repository. CI and review are covered on Diff to Deploy.
4. Replay tests#
Fault injection asks "what if this happens?". A replay test asks "did my change alter what happened before?". A replay test runs a recorded run again, with the same inputs, on new code, and compares the result with the recording. Any difference is either an intended change or a regression, and a person decides which.
Replay tests need determinism: the same inputs must give the same run. The example is built for it. The stand-ins are scripted or seeded, the clock is simulated, the ids are derived from the seed. The script below runs every case twice and compares the full run records, including events, spans, ids and stored rows.
import json
from run import case_mode, load_cases, run_case
same = 0
for case in load_cases():
recorded = json.dumps(run_case(case, case_mode(case)), sort_keys=True) # yesterday's run, stored
fresh = json.dumps(run_case(case, case_mode(case)), sort_keys=True) # today's code, same inputs
same += recorded == fresh
print(f"{same} of 18 runs reproduced exactly (events, spans, ids, store rows)")
18 of 18 runs reproduced exactly (events, spans, ids, store rows)
In this script both runs use today's code, so it proves determinism, the precondition. A real replay test stores the recording in the repository and compares against it after every change. With a real model the recording must include the model's outputs and the tool results, so the replay feeds them back instead of calling anything. That is the same data a durable workflow records anyway.
Common confusion. A replay test is not the replay that some engines use to restore a run after a crash, although both re-run workflow code over recorded results. One checks code changes; the other restores state. The durable workflows page explains the second.
5. Invariants#
Coverage and faults check specific paths. Some properties must hold on every path, whatever happens. An invariant is a statement that must be true for every run, at every moment, checked over all cases and modes rather than in one test.
The most important invariant of the quote workflow follows from the series' rule that the model recommends and a person decides: no human_decision exists unless a person's signal was applied, and exactly one exists if one was. The example checks it over every case in both modes:
def test_no_human_decision_without_a_persons_signal(self) -> None:
for cid in CASES:
for mode in ("dag", "loop"):
with self.subTest(case=cid, mode=mode):
r = run(cid, mode)
applied = [s for s in r["signals"] if s["action"] == "apply"]
self.assertEqual(r["human_decision"] is not None, bool(applied))
self.assertEqual(r["decisions_written"], len(applied))
That is test_orchestrator.py:test_no_human_decision_without_a_persons_signal, 36 subtests. A sibling test, test_apply_decision_refuses_without_a_signal, calls step 8 directly with no signal and expects it to refuse. A third, test_every_transition_is_legal, checks that every transition any run took is in the legal list, and every status is a known one.
Good invariants for an orchestrator are short and absolute:
| Invariant | Why it matters in the quote workflow |
|---|---|
No human_decision without an applied signal from a person | The model recommends; a person decides |
| No stored recommendation unless the guard passed in the same run | UW-008: the guard exists to stop the write |
| At most one stored row per run and revision | The idempotency key works (see durable workflows) |
| Every run that ends has a final status and a reason | No run disappears; failed runs reach the dead-letter list |
| Every transition taken is a legal one | The state machine is the contract of the orchestrator |
One trace_id per run when propagation is on | The record of a run is one piece (see tracing) |
What this is not. These tests check the orchestrator's behavior. How often the underwriting agent's recommendation is right, how good its reasons are and whether a release may ship are measured by evaluations: Eval Engineering. Checks around the model at run time belong to Guardrails Engineering.
6. Try it: cover the state machine#
The lab shows the states of a run. In coverage mode it starts from what the 18 cases exercised and lets you add the fault runs one by one, until no transition is left untaken. Click an event in any state: a legal one moves the run, an illegal one is refused with the reason.
The same information without JavaScript:
| Transition (from : event) | Taken by the 18 cases | Taken by a fault run |
|---|---|---|
new:start, running:step_succeeded, running:await_human, running:last_step_done | yes | |
running:step_failed (UW-008), running:cancel (UW-906), running:process_restarted (UW-901) | yes | |
suspended:resume_accepted, suspended:resume_stale (UW-902), completed:duplicate_signal (UW-905) | yes | |
suspended:signal_rejected | no | UW-004, an underwriter answers an escalation |
suspended:process_restarted, suspended:approval_timeout | no | UW-002, restart during the wait, then 24 hours of silence |
suspended:cancel | no | UW-001, the broker withdraws while the run waits |
Who decides here#
| Decision | Who |
|---|---|
| Which transitions are legal | Code: the state machine (RUN_TRANSITIONS) |
| Which faults to inject | People, from a written fault catalog per step and per wait |
| Whether a replay difference is a regression | A person, reviewing the difference |
| Which invariants must hold | People, from the business rules (R-HUMAN-01) and the design |
| What a model decides in these tests | Nothing; stand-ins replace it |
Checklist#
- Every model call and tool call can be replaced by a stand-in, and the clock by a simulated one.
- Transition coverage is measured, and a new transition without a test fails the build.
- A fault catalog lists, per step and per wait, the failures to inject; each row is a test.
- Recorded runs are replayed against new code, and differences are reviewed by a person.
- The invariant "no human decision without a person's signal" is checked over every case and mode.
- Every guard has a test in which it blocks, and the test checks that nothing was written.
- Tests of the orchestrator are kept apart from evaluations of the model, and both exist.
Coverage of steps, branches and combinationsRun transitions are the start, not the end
The run state machine is small: 14 transitions. Each step has its own (pending, running, succeeded, failed, skipped, compensated, and retry), and the workflow's branches multiply them: the fan-in can continue or route to request_info, the guard can pass, block or escalate, the resume can apply, ignore, reject or send the run back. Full combination coverage is impossible for any real workflow. A practical target is pairwise: every transition of every step taken at least once, and every pair of consecutive decisions on the main path (for example, fan-in result followed by guard result) taken at least once.
Coverage numbers are a map, not a score. The four transitions the cases missed were all on the wait, and that cluster told more than the percentage did.
Property-based testing of the orchestratorRandom event sequences against the invariants
Invariants become much stronger when the inputs are generated. A property-based test draws random sequences of events (signals with random tokens and roles, cancels at random moments, restarts after random steps, provider outcomes per attempt) and checks the invariants after each sequence. It finds orderings nobody wrote down, such as a cancel that arrives in the same step as the second click of a duplicate decision.
Two things make this practical. The orchestrator must accept faults as inputs (the example does, through run_case arguments), and a failing sequence must shrink to a small one a person can read. Libraries exist for this in most languages; which one is outside this page.
Testing loop modeSeeded stand-ins, many trials, and what to assert
In loop mode the stand-in model chooses tools from a seeded random plan, so one seed gives one path and twenty seeds give up to twenty paths (UW-010: nine distinct paths). Tests should not assert a path. They assert what must hold on every path: the budget is respected, every tool called is on the step's permission list, the guard runs after the loop, and the route is one the case allows. A path that skips the guidance search must end with the guard blocking, as one trial of UW-010 does.
The seed belongs in the test's name or output, so a failing trial can be run again alone. The same discipline applies with a real model: record the outputs of a failing trial, and the replay test turns it into a permanent regression test.
In one breath#
The orchestrator is ordinary code, so test it like code: replace the model, the tools and the clock with stand-ins, measure which transitions of the state machine your tests take, and inject the faults that business cases never reach, especially around the wait for a person. Replay recorded runs against new code. And check a few invariants over every case and mode, starting with the one that defines the series: no human decision without a person's signal.
What comes next#
You can now build the quote workflow, run it, break it on purpose and prove it behaves. A few problems remain that only appear at scale or with several agents acting at once: starting work before you know you need it, agents that wait for each other forever, runs that never get a worker, and judging a system in which no single agent owns the result. The next page is advanced.