The team behind the quote workflow has the three moves from Primitives: sequence, branch, loop. Now it has to write the workflow down so that an engine can run it, and a design review turns into an argument.
One engineer draws boxes and arrows: intake, three lookups side by side, the rules, the draft. "It is a graph." Another points at the run's status field: running, suspended, completed. "It is a state machine; the bug we had last month was a completed run that started again." A third says the run spends almost all its life waiting for the underwriter, so "it is event-driven: nothing happens until her decision arrives." And a fourth wants to give the model the tools and let it decide: "it is an agent loop."
Then someone runs case UW-010, which has the same input as UW-004 (application roof 9 years, provider 22), twenty times in each mode. The fixed graph takes the same path twenty times. The agent loop takes nine different paths, and one of the twenty runs fails. Same submission, same tools, same answers from the tools.
All four engineers are describing the same workflow correctly. They are writing down different things. Each of the four shapes answers one question about what runs next, and each one is silent, or wrong, about the others. This page takes them one at a time on the quote workflow, says what each cannot express, and ends with where each one fits. The agents here are scripted stand-ins and time is simulated, so these results show how the orchestration works, not how good or how fast any model is.
A state machine: which moves are legal now#
Start with the bug from last month (an illustrative incident, like every event in this example). A run had completed: the underwriter's decision was stored. Then a retried request from the underwriter cockpit arrived, and the run started again from the beginning. Nothing in the code said that a completed run is final.
A state machine says exactly that kind of thing. It is a table: the states something can be in, and for each state, which event moves it where. An event is something that happened: a step finished, a person decided, a cancellation arrived. A move that is not in the table is refused.
The run of the quote workflow has six states: new, running, suspended, completed, failed, cancelled. The last three are final. The table of legal moves is a list of 14 rows in core.py:RUN_TRANSITIONS, and one function looks a move up, core.py:run_transition:
def run_transition(state: str, event: str) -> dict[str, Any]:
"""Is `event` legal for a run in `state`? Returns legal, the new state and the reason."""
for t in RUN_TRANSITIONS:
if t["from"] == state and t["event"] == event:
return {"legal": True, "to": t["to"], "reason": t["reason"]}
if state in FINAL_RUN_STATUSES:
reason = f"A {state} run is final. Start a new run (a new run_id) instead."
else:
reason = _ILLEGAL_RUN.get(f"{state}:{event}", f"No transition from {state} on {event}.")
return {"legal": False, "to": None, "reason": reason}
The engine calls it on every status change (engine.py:Engine.set_run_status) and raises an error when a move is refused. The script below asks it about five moves, including last month's bug. A successful run proves that the table refuses those moves and says why; it does not prove the table is complete, which is what transition coverage in Testing checks.
import core
for state, event in [("suspended", "resume_accepted"), ("running", "resume_accepted"), ("suspended", "last_step_done"),
("completed", "start"), ("completed", "duplicate_signal")]:
t = core.run_transition(state, event)
print(f"{state:<10} + {event:<17} -> {'legal' if t['legal'] else 'refused'}: {t['to'] or '-'} · {t['reason']}")
suspended + resume_accepted -> legal: running · A valid decision arrived: right token, authorized role, same policy version.
running + resume_accepted -> refused: - · Only a suspended run can be resumed.
suspended + last_step_done -> refused: - · A waiting run cannot complete: the decision must arrive and apply_decision must run first.
completed + start -> refused: - · A completed run is final. Start a new run (a new run_id) instead.
completed + duplicate_signal -> legal: completed · The decision was already applied; the second signal is acknowledged and ignored.
The fourth line is last month's bug, refused. The fifth is the retried request that caused it, now legal and harmless: a completed run stays completed, and the signal is ignored. The third line is the property the insurer cares about most: a waiting run cannot complete without a person's decision.
Diagram: The run as a state machine · six states, and the events that move a run between them
Read left to right. Every run starts in new and ends in one of three final states. The arrows are the legal transitions of core.py, each caused by an event. A move that is not drawn, such as resuming a running run, is refused with a reason. Moves that stay in the same state (a duplicate signal, a restart) are listed in the table below the diagram.
- Before
- new: The run exists only as an id and an input. Event start moves it to running.
- Doing work
- running: Steps run one after another. Each step_succeeded keeps the run here until a step suspends, fails or ends it.
- Waiting
- suspended: The run holds no compute and waits for an event: the underwriter's decision, a cancellation, a timeout.
- Final
- completed: The last step finished. A second decision signal is acknowledged and ignored.
- failed: A step failed with no fallback, or the guard blocked the write. Final.
- cancelled: A cancellation arrived; finished steps with a compensation were undone. Final.
In the lab, click events on a run and watch legal moves happen and illegal ones refused with their reason; a counter shows which moves you have exercised. The table after it lists all 14 legal moves.
| From | Event | To | Why |
|---|---|---|---|
new | start | running | A run starts at its first step |
running | step_succeeded | running | A step finished; the transition rules choose the next step |
running | await_human | suspended | The run stops holding compute and waits for the underwriter |
running | last_step_done | completed | The last step finished |
running | step_failed | failed | A step failed with no fallback route, or the guard blocked the write |
running | cancel | cancelled | Finished steps with a compensation are undone |
running | process_restarted | running | The run continues from its last checkpoint |
suspended | resume_accepted | running | Right token, authorized role, same policy version |
suspended | resume_stale | running | The policy version changed; the rules are checked again |
suspended | signal_rejected | suspended | Wrong token or a role that may not decide; keeps waiting |
suspended | approval_timeout | suspended | No answer in time; escalated to a senior underwriter |
suspended | process_restarted | suspended | The waiting run is read back from its checkpoint |
suspended | cancel | cancelled | Finished steps with a compensation are undone |
completed | duplicate_signal | completed | Already applied; acknowledged and ignored |
What a state machine cannot express well is work that happens at the same time. While the three lookups run, each can be waiting, done or failed. As states of one machine, that is every combination of the three, and every combination needs its own rows. Nor does the table say what one step needs from another: nothing in it says the rules need the roof assessment. For that the team needs a second shape.
For a backend engineer. This is the status column of an order table with its allowed transitions, enforced in one place instead of in every service that writes the column. The comparison holds completely: the run's state machine has nothing to do with models.
A DAG: what must finish before what#
The engineer with boxes and arrows was answering a different question: which steps need which. The rules need all three lookups. The three lookups need only the submission, not each other.
A DAG, a directed acyclic graph, writes that down. Each step is a node. Each arrow means "must finish before". "Acyclic" means there is no cycle: no path of arrows that leads back to a step already passed. Steps with no path between them may run at the same time, so a DAG says, without extra words, that the three lookups can run in parallel and the rules must wait for all of them.
Pipeline mode of the example (--mode dag) is that graph. The script below reads the transitions of both modes of workflow.py:build_workflow and checks that each forms a graph with no cycle, by removing steps that nothing still points to until none are left. It was written for this page and run against the example. A successful run proves that neither workflow definition contains a way back. It does not prove that a run never repeats work; the next paragraphs show where repetition lives instead.
from workflow import build_workflow
for mode in ("dag", "loop"):
wf = build_workflow(mode)
edges = {name: sorted({t.to for t in step.transitions if t.to}) for name, step in wf.steps.items()}
seen, order = set(), []
while len(order) < len(edges):
ready = [n for n in edges if n not in seen and all(n not in edges[m] for m in edges if m not in seen)]
if not ready:
print(mode, "has a cycle")
break
seen.add(ready[0])
order.append(ready[0])
print(f"{mode:<4} acyclic: {len(order) == len(edges)} · {' > '.join(order)}")
dag acyclic: True · intake > gather_evidence > check_eligibility > draft_recommendation > verify_process > record_recommendation > await_underwriter > apply_decision
loop acyclic: True · intake > agent_loop > verify_process > record_recommendation > await_underwriter > apply_decision
Both definitions are acyclic, and that buys two things. Every run of the graph ends. And every possible path can be listed before any run starts, which is why UW-010 in pipeline mode takes one path in all 20 trials:
UW-010 · dag mode · 20 trials · 1 distinct path(s)
status: suspended 20 · route: escalate 20 · calls per run: 7 to 7
The price is the thing a DAG cannot express: going back. The quote workflow needs to go back in two places. A failed lookup is tried again. And this site's own case UW-902 sends a waiting run back to the rules when the policy fixture changes while the underwriter is deciding. Neither is an arrow in the graph. The retry lives inside the gather_evidence step. The return to the rules is done by the engine (engine.py:Engine._go_back_stale), outside the graph, through the state machine's resume_stale event, and it runs the later steps again as revision 2. The graph stays acyclic because the cycles were moved somewhere bounded.
Common confusion. "We use a DAG, so we cannot retry." You can; the retry lives inside a node, with its own bound. What a DAG forbids is an arrow back to an earlier node. Spring Batch draws the same line: its flows are transitions between steps, and retrying a failing item is configured on the step itself (faultTolerant().retryPolicy(...) in its version 6 documentation).
Event-driven: what reacts when something arrives#
The third engineer looked at time. In UW-004 the work of the run takes a few simulated seconds and the wait for the senior underwriter takes about three simulated hours. In real life the wait could be days. For almost all of its life, the run is not doing anything. It is waiting for something to arrive.
Event-driven control is written as reactions: when this event arrives, do that. Nothing polls. The run waits, and the event moves it. In the quote workflow three outside events can arrive at a waiting run: the underwriter's decision (engine.py:Engine.signal), a cancellation from the broker (Engine.cancel), and an approval timeout (Engine.check_approval_timeout). When the decision arrives, a pure function decides what it does (core.py:accept_resume): apply it, send the run back because the policy changed, ignore a duplicate, or reject a wrong token or a role that may not decide. In the UW-004 run on the Start page, that is the line signal: senior_underwriter approve -> apply.
What event-driven control cannot express on its own is the whole order. If every step were only a handler reacting to the previous step's event, the rule "the guard runs before the write" would live in no single place: it would be a consequence of which handler happens to listen to which event. Nobody could read the workflow, and no test could check its transitions. So the example uses events only at the edges of a run: to start it, to resume it, to stop it. The order between those edges stays in the workflow. Human approvals is the home of the resume, with its token and its stale-policy check.
What this is not. This page is about which events may move a run. The format of the messages that carry events between agents and services belongs to Agent Communication, and the log of events as evidence of what happened belongs to Observability.
An agent loop: the model picks the next step#
The fourth engineer wanted the model to decide. In loop mode (--mode loop), steps 2 to 4 of the quote workflow become one step, agent_loop. Inside it an agent loop runs: the model is called, it names the next tool, the program runs that tool and hands back the result, and the model is called again, until it says finish or a limit stops it. The choice of what runs next has moved from code to a model, inside one step.
This is the step, workflow.py:step_agent_loop, quoted whole. Read it for what stays in code. The budget is checked before every turn. The model only names a tool; run_loop_tool runs it, after a permission check. And the draft at the end is checked by the guard in the next step, verify_process. A successful run of a loop proves that it ended for a recorded reason; it does not prove that it called the right tools.
def step_agent_loop(ctx: StepContext) -> dict[str, Any]:
"""Loop mode. The stand-in model picks the next tool each turn; the budget, in code, ends the loop."""
inp, run = ctx.run["input"], ctx.run
sub = ctx.output("intake")["submission"]
limits = {**core.LOOP_LIMITS, **inp.get("budget", {})}
model = agents_sim.LoopModel(inp.get("loop_behaviour", "careful"),
random.Random(f"{inp['seed']}:{inp['case_id']}:{inp['trial']}"))
state: dict[str, Any] = {"evidence": {}, "eligibility": None, "searches": 0}
turns: list[dict[str, Any]] = []
run["budget_used"] = used = {"turns": 0, "tool_calls": 0, "tokens": 0, "wall_clock_ms": 0}
termination = None
while termination is None:
termination = core.budget_check(used, limits)
if termination:
ctx.event("budget_exhausted", {"reason": termination, "used": dict(used), "limits": limits})
break
turn = used["turns"] + 1
action, cost = model.next_action(turn), agents_sim.LoopModel.turn_cost(turn)
t0 = run["clock_ms"]
ctx.tick(cost["ms"])
ctx.span("loop_model_turn", "model", t0, run["clock_ms"], turn=turn, chose=action["tool"], stand_in=True,
simulated_tokens=cost["tokens"])
calls = 0 if action["tool"] == "finish" else run_loop_tool(ctx, action, sub, state)
usage = {"tool_calls": calls, "tokens": cost["tokens"], "wall_clock_ms": run["clock_ms"] - t0}
used.update(turns=turn, tool_calls=used["tool_calls"] + calls, tokens=used["tokens"] + cost["tokens"],
wall_clock_ms=used["wall_clock_ms"] + usage["wall_clock_ms"])
turns.append({"turn": turn, "tool": action["tool"], **usage})
if action["tool"] == "finish":
termination = "finished"
run["termination_reason"] = termination
draft = loop_draft(ctx, sub, state, termination)
return {"turns": turns, "termination_reason": termination, "limits": limits, "budget_used": dict(used),
"evidence": state["evidence"], "eligibility": state["eligibility"], "draft": draft,
"facts": facts_from(sub, state["evidence"]) if state["evidence"].get("roof_assessment") or state["eligibility"] else None}
Now the twenty trials of UW-010 in loop mode. The stand-in model follows a seeded random plan per trial (agents_sim.py:LoopModel, seed 7), so the variation is the same on every machine; a real model varies for its own reasons and by its own amount.
cd examples/uw-orchestrator
/opt/homebrew/bin/python3 run.py --case UW-010 --mode loop --trials 20
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-010 · loop mode · 20 trials · 9 distinct path(s)
status: suspended 19, failed 1 · route: escalate 20 · calls per run: 8 to 13
x4 get_roof_assessment > search_guidance > check_eligibility > finish
x4 search_guidance > get_roof_assessment > find_similar_decisions > check_eligibility > finish
x2 find_similar_decisions > search_guidance > get_roof_assessment > check_eligibility > finish
x2 find_similar_decisions > get_roof_assessment > search_guidance > check_eligibility > finish
x2 get_roof_assessment > search_guidance > find_similar_decisions > check_eligibility > finish
x2 search_guidance > get_roof_assessment > check_eligibility > finish
x2 get_roof_assessment > find_similar_decisions > search_guidance > check_eligibility > finish
x1 find_similar_decisions > get_roof_assessment > search_guidance > search_guidance > check_eligibility > finish
x1 get_roof_assessment > find_similar_decisions > check_eligibility > finish
wrote 20 run file(s) in out/runs/ and out/summary.json
Nine paths, all ending in escalate. The order of lookups changes, the optional find_similar_decisions is sometimes skipped, the guidance is once searched twice. Most of that variation is harmless. The last path is not: it never calls search_guidance, so the draft cites a rule that no search returned, and the guard blocked the write. That is the one failed trial. Case UW-008 makes the same skip on purpose, on the facts of UW-001:
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-008 · Right recommendation, required check skipped · loop mode · run-e220 · trace e9d1d3c00fc3...
path: intake > agent_loop > verify_process
loop: get_roof_assessment > find_similar_decisions > check_eligibility > finish · ended: finished · turns 4 · tool calls 3 · tokens 3900
guard: block (The draft cites R-ROOF-01, but no guidance search returned it.)
recommendation: none stored (rows written: 0)
human_decision: none
status: failed · model calls 5 · tool calls 3 · simulated time 4.4 s
expected: as stated in the case
wrote 1 run file(s) in out/runs/ and out/summary.json
The draft said decline, citing R-ROOF-01, which is the right answer for a 31-year-old roof in wind zone C. It was right without having read the rule. In pipeline mode the same case retrieves the guidance as a fixed step and the run suspends normally for the underwriter. This is what an agent loop cannot express: a promise that a required step happens. The path is chosen while the run goes on, so "always search the guidance before citing a rule" cannot be written into the loop. It has to be checked after it, in code.
The same idea in a coding agent. A coding agent's loop picks which file to read and which command to run next, and its path differs from one issue to the next. A team that requires the tests to run before a pull request does not trust the loop to remember: CI runs them, as a fixed step outside the loop. That is the same move as verify_process.
The four shapes side by side#
Each shape answers one question, and each is silent about the question another answers.
Diagram: Four shapes of control · what each writes down, and what it cannot express
Read each column on its own: one shape, the question it answers, and the thing it cannot say. Left to right, the decision about what runs next moves from a table you wrote, to a graph you wrote, to events from outside, to a model's choice. The quote workflow uses all four, each where it fits.
- 1 · State machine
- States and legal moves: Answers: from this state, on this event, where may the run go? Any move not in the table is refused.
- Cannot express well: Work that runs at the same time multiplies the states; what each step needs from another is not in the table.
- 2 · DAG
- Steps and dependencies: Answers: which steps must finish before this one may start? Steps with no path between them may run at the same time.
- Cannot express: A cycle: no way back to an earlier step. A retry or a revision must live inside a step or outside the graph.
- 3 · Event-driven
- Reactions to events: Answers: what happens when something arrives from outside, at a time nobody chose?
- Cannot express alone: The overall order lives in no single place; it emerges from many handlers, so it is hard to see and to test.
- 4 · Agent loop
- A model picks the next step: Answers: which tool next, given what the model has seen so far? The path is chosen while the run goes on.
- Cannot promise: Which steps will run, and in which order, is not known in advance; a required step may be skipped.
| Shape | Writes down | Question it answers | Cannot express | Fits in the quote workflow |
|---|---|---|---|---|
| State machine | States and legal moves | Is this move allowed now? | Parallel work; what a step needs from another | The status of a run and of each step |
| DAG | Steps and dependencies | What must finish before this starts? What may run at once? | Going back to an earlier step | The order of the steps in pipeline mode |
| Event-driven | Reactions to events | What happens when this arrives? | The whole order, in one place | The underwriter's decision, a cancellation, a timeout |
| Agent loop | Nothing fixed; a model chooses | What is the best next step, given what we have seen? | A fixed path; a required step | Optionally, the evidence lookups inside one step |
Read the last column top to bottom. The quote workflow is not one shape. Its runs are governed by a state machine, its steps are ordered by a DAG, its waits end on events, and in loop mode one of its steps holds an agent loop. The question for a design review is not "which shape is our workflow?" but "which shape governs which decision?".
The lab compares the last two side by side. Pick a case, the loop's turn budget and, for UW-010, a trial; it shows both paths, the number of calls, and whether the required verification happened. The table after it holds the results for three cases at three budgets, from run.py:run_case with seed 7.
| Case | Mode | Turn budget | Run status | Route | Loop ended by | Guard | Model calls | Tool calls |
|---|---|---|---|---|---|---|---|---|
UW-004 | pipeline | completed | escalate | pass | 3 | 4 | ||
UW-004 | loop | 3 | completed | escalate | budget_exhausted:turns | pass | 4 | 4 |
UW-004 | loop | 6 | completed | escalate | finished | pass | 6 | 5 |
UW-008 | pipeline | suspended | decline | pass | 3 | 4 | ||
UW-008 | loop | 3 | suspended | escalate | budget_exhausted:turns | pass | 4 | 4 |
UW-008 | loop | 6 | failed | decline (draft, not stored) | finished | block | 5 | 3 |
UW-904 | pipeline | suspended | approve | pass | 3 | 4 | ||
UW-904 | loop | 6 | suspended | escalate | budget_exhausted:turns | pass | 7 | 7 |
UW-904 | loop | 8 | suspended | escalate | budget_exhausted:turns | pass | 9 | 9 |
Three readings. With a budget of 3 turns the loop never finishes, so code writes an escalation; UW-004 reaches the right route for a reason that has nothing to do with the roof. For UW-008 a larger budget makes things worse, not better: the loop finishes, skips the guidance and the guard blocks. And for UW-904, whose stand-in keeps searching, a larger budget only buys more calls before the same ending. In every row a run status or a route came from code: the budget, the guard, or the fixed steps.
Who decides what runs next here#
Each shape has a different answer. In a state machine, code decides which moves are legal, and an event chooses among them. In a DAG, code decides the order, completely, before the run starts. In event-driven control, whoever sends the event decides when; in the quote workflow the person who matters is the underwriter, whose decision is the only event that can end a wait with a decision. In an agent loop, the model decides the next step, and code decides how many steps it gets and whether its result may be stored. The model recommends; a person decides, and none of the four shapes lets a model write that decision.
Frameworks usually lead with one shape and offer the others as features: a graph you declare with conditional edges, a durable workflow that waits for signals, a provider SDK that runs an agent loop with a turn limit, statuses with checked transitions. When you read one, ask which shape it makes easy and where it puts the other three. Frameworks reads the products on their own documentation.
Checklist#
- The run has an explicit state machine, and every status change goes through it.
- A move that is not in the table is refused with a reason, not silently applied.
- The workflow graph has no unplanned cycle; every way back (retry, revision) is bounded and recorded.
- Steps that do not depend on each other are not forced into a sequence.
- Outside events only start, resume or stop a run; the order of steps is written in one place.
- Every agent loop has a budget in code and a guard in code at its exit.
- Every required step is either a fixed step or checked by a guard after the loop.
- For each decision, the team can name the shape that governs it.
Why a state machine is the wrong place for parallel workThree lookups with four outcomes each is 64 combinations
If each of the three lookups can be pending, running, done or failed, a single state machine for the whole step needs a state for each combination: 4 × 4 × 4 = 64, before counting retries and lateness. Every new lookup multiplies the table again. This is the classic state explosion, and it is why the run's state machine stops at the level of the run and each step's own status machine stays small.
The example handles the parallel part with a different tool: core.py:simulate_fanout runs the lookups on workers, and core.py:fan_in reduces their outcomes to one decision (continue or request_info) under a rule that names which lookups are required. The state machine then sees one step that succeeded with a decision. Hierarchical state machines (states that contain machines) are the other classic answer; they keep the table small at each level at the cost of a second level to reason about. Concurrency is the home of fan-out and fan-in.
One path is not the same as the right pathWhat the fixed pipeline guarantees, and what it does not
UW-010 takes one path in all 20 pipeline trials. That is a property of the shape: code decides every next step, and the stand-in agents are scripted. It says that the run is repeatable in its order of steps. It says nothing about whether the recommendation is right; a wrong rule applied the same way twenty times is wrong twenty times.
With a real model inside the drafting step, a fixed pipeline still varies in what the model writes, even though it never varies in which steps run. That is why the guard in verify_process checks the draft against the rule outcome and the retrieved rules, in both modes. Measuring how often a recommendation is right is a question for evals, owned by Eval Engineering; this page claims only that the pipeline's path is fixed by design.
The loop's variation in this example is also by design: a seeded random plan, the same on every machine. It shows that a path chosen by a model can vary and what the orchestration does about it. It does not estimate how much a real model would vary; that depends on the model, its settings and its inputs, and a lower randomness setting makes a model less variable, not deterministic.
Event-driven control and event sourcingTwo ideas with similar names
Event-driven control means events move the run. Event sourcing means the state is stored as the list of events that produced it, and rebuilt by replaying them. They are independent: a system can have either without the other.
The example is event-driven at its edges and keeps an event log, but it is not event-sourced. It saves the whole run state in a checkpoint after every step and restores from that copy, not by replaying events (the README of the example lists event-history replay as not implemented). Some durable workflow engines do the opposite: they record each step's result in a history and rebuild a run by replaying the workflow code against it. Durable workflows compares the two ways of surviving a crash, and Frameworks says which product does which, as read on its documentation.
In one breath#
A state machine says which moves are legal now, a DAG says what must finish before what, event-driven control says what happens when something arrives, and an agent loop lets a model choose the next step as it goes. Each cannot say what another says: a DAG cannot go back, a state machine drowns in parallel work, events alone hide the order, and a loop cannot promise that a required step happens. So the quote workflow uses each where it fits: a state machine for the run, a DAG for the steps, events for the person's decision, and an agent loop only inside one bounded step with a guard at its exit. In twenty trials the graph took one path and the loop took nine, and the one bad path was stopped by code, not by the loop.
You can now write the quote workflow down in four ways and say which part each one governs. Loop mode raised a question this page did not answer: it replaced three steps and three agents with one agent and four tools. Was that a simplification, or did it lose something the three agents gave? One or many takes that question: when one agent with tools is enough, and what a second agent really adds and costs.