Run run-9a32 is working on case UW-901, this site's own case with the facts of UW-004: the application says the roof is 9 years old, the roof-data provider says 22. The orchestrator has finished step 2. Three lookups went out, three answers came back, the evidence is in memory. Then the machine is replaced during a routine deploy, and the orchestrator process dies.
What should happen next?
The cheap answer is "start the run again". For this case it would even give the same result: the stand-in provider would say 22 again, the rules would say R-ROOF-03 again. But in production every lookup costs money and time, a provider may answer differently tomorrow, and later steps have effects in the world. If the process had died after step 6, record_recommendation, starting again would store a second recommendation for the same submission. The underwriter cockpit would show two rows. One of them might be approved and the other declined.
So the real question has two halves. How does a run continue from where it was, instead of from the start? And when a step must run again, how do we make sure its effect in the world happens only once? This page answers both, and then a third question that follows from them: when a run must stop on purpose, what happens to the work it already did?
One note before any output: the agents here are scripted stand-ins and time is simulated, so these results show how the orchestration works, not how good or how fast any model is.
1. Checkpoints: write down where the run stands#
A checkpoint is a saved copy of where the run stands, written after a step finishes. After step 2 of run-9a32, the orchestrator writes down that intake and gather_evidence succeeded, their outputs (the submission and the three pieces of evidence), and which step comes next. A new process that reads this record can continue with step 3.
The code in the example is short, because the hard part is deciding when to call it, not how:
def checkpoint(self, run: dict[str, Any], after_step: str | None) -> None:
"""Save the whole run state. A restart continues from here."""
self.event(run, "checkpoint_saved", after_step)
self.services.checkpoints.save(run)
That is engine.py:Engine.checkpoint. The engine calls it after every step and when a run suspends. The store writes one JSON file per run. Reading it back is Engine.restore:
def restore(self, run_id: str) -> dict[str, Any]:
"""Read a run back from its checkpoint (a new process). Finished steps are not run again."""
run = self.services.checkpoints.load(run_id)
self.runs[run_id] = run
if run["status"] in core.FINAL_RUN_STATUSES:
return run
run["clock_ms"] += RESTART_GAP_MS
run["restarts"] += 1
finished = [n for n in self.wf.order if run["steps"][n]["status"] == "succeeded"]
run["restored_from_checkpoint"] = finished
self.set_run_status(run, "process_restarted")
self.event(run, "run_restored", None, {"from_checkpoint": finished, "next_step": run["current_step"]})
return run
The two commands below are two separate Python processes. The first one runs UW-901 and dies after the checkpoint of step 2. The second one starts cold, knows only the run_id, and continues.
$ $PY run.py --case UW-901 --crash-after gather_evidence
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-901 · dag mode · run-9a32: orchestrator process died after checkpoint of gather_evidence
checkpoint: steps succeeded intake, gather_evidence · next step check_eligibility
resume with: python3 run.py --resume run-9a32
$ $PY run.py --resume run-9a32
UW-901 · The orchestrator process dies after the evidence is gathered · dag mode · run-9a32 · trace 76ddf14ebd87...
crash: process died before this resume; restored from checkpoint: intake, gather_evidence
path: intake > gather_evidence > check_eligibility > draft_recommendation > verify_process > record_recommendation > await_underwriter
fan-in: roof_assessment ok [ok], guidance ok [ok], similar_decisions ok [ok]
guard: pass (Eligibility ran and every cited rule was retrieved.)
recommendation: escalate · cites R-ROOF-03 · missing - · policy 2026.2 · stored row status stored
human_decision: none yet (waiting for: senior_underwriter)
status: suspended · model calls 3 · tool calls 4 · simulated time 33.1 s
expected: as stated in the case
What this proves: the resumed run made 4 tool calls, the same number as a run that never crashed, so the three lookups were not made again. It kept the same trace_id, so the record of what the run did is one piece. The 30 extra simulated seconds are the restart gap the example adds. What it does not prove: that the checkpoint store itself survives a disk failure. Where and how checkpoints are stored durably is the subject of Memory Engineering.
A durable workflow is a workflow whose runs survive the death of the process that runs them, because the state lives outside that process. That is the whole definition. Everything else on this page is about making the survival correct.
For a backend engineer. Spring Batch keeps a JobRepository with the execution context of each step, and a restarted job skips the steps that completed. The checkpoint here is the same idea at the granularity of one workflow step. Where the comparison stops: a batch job usually restarts by an operator's command, while a workflow engine restarts a run on its own, and a workflow step may wait for a person for days.
2. The gap between the work and the checkpoint#
A checkpoint after every step sounds complete. It leaves one gap, and the gap is where the trouble lives. A step does its work, then the checkpoint is written. If the process dies between those two moments, the work happened but the checkpoint does not know it. On resume, the step runs again.
For a read, that is harmless: the roof assessment is fetched twice. For a write, it is the duplicate row from the opening. The example can crash exactly there, with --crash-before-checkpoint:
$ $PY run.py --case UW-901 --crash-after record_recommendation --crash-before-checkpoint
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-901 · dag mode · run-9a32: orchestrator process died before checkpoint of record_recommendation
checkpoint: steps succeeded intake, gather_evidence, check_eligibility, draft_recommendation, verify_process · next step record_recommendation
resume with: python3 run.py --resume run-9a32
The recommendation was stored, and the checkpoint still says step 6 is next. The resume will run step 6 again. Whether that stores a second row depends on one thing: the key.
Diagram: The gap between the work and the checkpoint · where a repeat comes from
Read left to right. Step 6 stores the recommendation, then the checkpoint is written. A crash between the two leaves a stored row that the checkpoint does not know about. On resume the step runs again; the idempotency key turns the repeat into no change. The agents are scripted stand-ins and time is simulated.
- 1 · Before the step
- Checkpoint: The saved state says steps 1 to 5 succeeded and step 6 is next.
- 2 · The step does its work
- record_recommendation: The write reaches the store with the key run-9a32:record_recommendation.
- 3 · The gap
- Process dies: The work happened, the record of it did not. This moment always exists, however fast the checkpoint is.
- 4 · A new process resumes
- Restore: Engine.restore reads the checkpoint. Finished steps are not run again; step 6 is.
- Write again: The store finds a row with the same key and returns it. With the key off, a second row appears.
3. The idempotency key: make the write safe to repeat#
An operation is idempotent when doing it twice has the same effect as doing it once. Setting a field to 5 is idempotent. Appending a row is not. The write in the quote workflow appends a row, so the workflow makes it idempotent from the outside: it sends a key with it.
An idempotency key is a value the caller builds once and sends with every repeat of the same write, so the receiver can recognize the repeat. Here the key is run-9a32:record_recommendation: the run_id plus the step name. The store keeps the key with the row, and when the same key arrives again it returns the existing row instead of adding one.
def put_recommendation(self, row: dict[str, Any], idempotency_key: str | None) -> dict[str, Any]:
"""Store a recommendation unless a row with this key exists. Returns the row and whether it is new."""
self.data["write_attempts"] += 1
if idempotency_key:
for existing in self.data["recommendations"]:
if existing["idempotency_key"] == idempotency_key:
return {"row": existing, "created": False}
row = dict(row, row_id=len(self.data["recommendations"]) + 1, idempotency_key=idempotency_key, status="stored")
self.data["recommendations"].append(row)
self.save()
return {"row": row, "created": True}
That is engine.py:RecordStore.put_recommendation. The key is built by workflow.py:idempotency_key, which adds the revision after a stale approval (see human approvals), because a new revision is a new recommendation and must get its own row.
The following script crashes UW-004 in the gap, once with the key and once without. It ran in a scratch copy of the example.
from run import load_cases, new_services, run_case
case = load_cases("UW-004")[0]
crash = ("record_recommendation", "before_checkpoint") # the write happened; its checkpoint did not
for use_key in (True, False):
services = new_services(case)
services.use_idempotency_key = use_key
r = run_case(case, "dag", services=services, crash_at=crash)
print(f"idempotency key {'on ' if use_key else 'off'} -> rows written {r['rows_written']}, status {r['status']}")
idempotency key on -> rows written 1, status completed
idempotency key off -> rows written 2, status completed
Both runs complete. Both look healthy in a dashboard. Only one of them left the store correct. The event log of the first one shows the difference in one line: write_deduplicated with row_id 1.
Common confusion. The key must be stable across repeats. A key built from the time, or a new random value per attempt, makes every repeat look new and protects nothing. Built from the run_id and the step name, it is the same in every process that ever runs this step of this run.
The same idea in a coding agent. A coding agent that pushes a branch and opens a pull request can crash between the two. On resume, "open a pull request for this branch" must find the existing one instead of opening a second. The branch name plays the role of the key. The pull request itself is the agent's durable state: it survives the agent. Review and CI around it are covered on Diff to Deploy.
4. Retries and the attempt number#
Some failures are not crashes of the orchestrator but failures of one step: the roof-data provider times out. The orchestrator tries the step again. A retry is a new try of the same step inside the same run, made because the previous try failed in a way that might not happen again.
Each try gets an attempt number: 1 for the first try, 2 for the first retry. In UW-006 the roof lookup has attempt_number 1 and 2, both time out, and then the fan-in rule sends the run down the fallback route to request_info. The two tries are two spans in the same trace. The run_id does not change, because it is still the same run. Keeping these apart is the job of the tracing page.
Retries are where idempotency stops being optional. A step that times out may have succeeded on the other side: the provider did the work, and the answer was lost on the way back. Retrying a read is safe. Retrying a write without a key is the duplicate row again.
For a backend engineer. A@Retryablemethod (Spring Framework 7 has the annotation built in) retries inside one process. If the process dies, the retry count dies with it. A durable workflow keepsattempt_numberin the run state, so a retry budget of two attempts is still two after a restart.
5. Replay: the other way to restore#
The example restores a run by reading its saved state: the outputs of the finished steps. Some engines restore a run differently. They keep the history of events (step 1 started, step 1 returned this value, step 2 started) and rebuild the state by running the workflow code again from the top, feeding it the recorded results instead of calling the steps. That is replay.
Replay buys a complete history and lets the workflow be ordinary code with loops and variables. It costs a strict rule: the workflow code must make the same decisions every time it is replayed. A workflow that reads the clock, draws a random number or calls a model directly, outside a recorded step, decides differently on replay and the engine can no longer match the history. In the quote workflow, the stand-in model's choice in loop mode would have to be a recorded step, never a call inside the workflow code. The example does not implement replay; it resumes from saved state, and the page says so. Engines that replay are described on Frameworks.
6. Compensation and cancellation#
Case UW-906 is this site's own. The recommendation has been stored. Then the broker writes: the client has withdrawn the request. The run must stop.
Stopping is easy. The harder question is the row in the store. It says "approve, stored" for a request that no longer exists, and an underwriter might act on it. A database rollback cannot help: the write committed long ago, in another transaction.
A compensation is a step that undoes the business effect of an earlier finished step. It does not erase history. For step 6 the compensation marks the stored recommendation as withdrawn:
def compensate_record(ctx: StepContext) -> dict[str, Any]:
"""Compensation of step 6: the stored recommendation is marked withdrawn (not deleted)."""
row_id = ctx.output("record_recommendation")["row_id"]
ctx.services.store.mark_recommendation(row_id, "withdrawn")
return {"row_id": row_id, "status": "withdrawn"}
A cancellation is a request to stop a run on purpose. In the example (Engine.cancel), it is checked at step boundaries, and a waiting run is cancelled at once. Engine.finish_cancel then walks the finished steps in reverse order, runs the compensation of each step that has one, marks the remaining steps skipped, and ends the run with status cancelled. The real run:
$ $PY run.py --case UW-906
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-906 · The broker withdraws the request after the recommendation is stored · dag mode · run-8fe1 · trace 7eda2941729b...
path: intake > gather_evidence > check_eligibility > draft_recommendation > verify_process > record_recommendation
fan-in: roof_assessment ok [ok], guidance ok [ok], similar_decisions ok [ok]
guard: pass (Eligibility ran and every cited rule was retrieved.)
recommendation: approve · cites - · missing - · policy 2026.2 · stored row status withdrawn
human_decision: none
status: cancelled · model calls 3 · tool calls 4 · simulated time 3.1 s
expected: as stated in the case
wrote 1 run file(s) in out/runs/ and out/summary.json
The stored row now says withdrawn. Nobody decided anything, and the record says so: human_decision is none.
Diagram: Cancellation with compensation · UW-906
Read left to right. The run stored an approve recommendation. The broker withdraws. The cancel is checked at the next step boundary; finished steps with a compensation are undone in reverse order; the rest are skipped. Nothing is deleted.
- 1 · Done
- Steps 1 to 6: The recommendation approve is stored as row 1 with status stored.
- 2 · Outside event
- Broker withdraws: Engine.cancel records the request; the engine acts on it at the next step boundary.
- 3 · Undo the effect
- compensate_record: The business effect of step 6 is undone by changing the row's status, not by deleting it.
- 4 · End
- Run cancelled: Either every step finishes, or the finished ones are compensated. No human decision is recorded.
A saga is the pattern behind this: a long piece of work written as a sequence of steps, each with a compensation, so that either all steps finish or the compensations of the finished ones run. The word comes from a 1987 database paper by Garcia-Molina and Salem. Its abstract describes exactly this guarantee: either all the transactions in a saga complete, or compensating transactions amend the partial execution. Among its examples of long-lived transactions is the processing of claims at an insurance company.
What this is not. Compensation is not a rollback, and a saga gives no isolation: between step 6 and its compensation, another reader could see the stored row. How the platform guarantees delivery of the cancel request, and the wider subject of recovering from failure under load, belong to Agent Reliability Engineering.
7. Try it: kill the process#
The lab crashes a run with the facts of UW-004 (the same facts as UW-901). Pick the step after which the process dies, and whether it dies before or after that step's checkpoint. Switch the idempotency key on or off. Then resume, and see which steps come from the checkpoint, which run again, and how many rows the write leaves. A second control cancels the run after the write and shows the compensation.
The same information without JavaScript, from the example's runs:
| What happens | Steps read from the checkpoint | Steps run again | Rows stored |
|---|---|---|---|
Dies after the checkpoint of gather_evidence | intake, gather_evidence | none; the run continues at check_eligibility | 1 |
| Dies after the write, before its checkpoint, key on | steps 1 to 5 | record_recommendation (deduplicated) | 1 |
| Same, key off | steps 1 to 5 | record_recommendation (a second row) | 2 |
Cancelled after the write (UW-906) | not a crash | none; the write is compensated | 1, status withdrawn |
Who decides here#
| Decision | Who |
|---|---|
| When to save a checkpoint | Code: after every step and at every wait |
| Which steps to skip on resume | Code: the checkpoint says which succeeded |
| Whether to retry a failed step, and how often | Code: a retry policy with an attempt limit |
| Whether a repeated write stores a row | Code: the store, from the idempotency key |
| To cancel a run | An outside event: here the broker's withdrawal, applied by code |
| What a model decides here | Nothing. A model's output is a step result that the checkpoint saves |
Checklist#
- State lives outside the process: a crash at any point loses at most the step in flight.
- Every step is either safe to run twice, or its effect carries a stable idempotency key.
- The key is built from the run and the step (and the revision), never from the time or a fresh random value.
attempt_numberis part of the run state and survives a restart; retries have a limit.- Every step with an effect in the world has a written compensation, or a written reason why it needs none.
- Cancellation is checked at step boundaries and runs the compensations in reverse order.
- A test crashes the run before and after the checkpoint of each step and checks the store.
Exactly-once effects, preciselyAt-least-once execution plus deduplication
No checkpointing scheme gives exactly-once execution of a step: there is always a moment between doing the work and recording it. What a durable workflow offers is at-least-once execution with recorded results, and the effect becomes exactly-once only where the receiver deduplicates. That is why the key travels with the write to the store, and is checked there, in the same place the row is written. A check in the orchestrator ("did I already write this?") followed by a write is a race between two processes. The check and the write must be one atomic operation at the receiver: a unique constraint on the key column, or a conditional insert.
How long the receiver remembers keys matters too. The example keeps them forever. A real store keeps them for a window, and the window must be longer than the longest time a run can take to retry or resume, including the days a run can wait for an underwriter.
Designing compensationsSemantic undo, ordering, and what cannot be undone
A compensation is a business decision, not a technical one. Marking a recommendation withdrawn is right because the cockpit understands that status and an auditor can see both the row and its end. Deleting the row would be a technical undo that destroys evidence.
Compensations run in reverse order because later steps may depend on earlier ones. They must themselves be idempotent, because the cancellation can be interrupted and resumed like any other part of the run. And some effects have no compensation at all: an email already sent to the broker can only be followed by a correction. Such steps belong as late as possible in the workflow, after the last point where the run can still be cancelled cheaply.
A compensation can also fail. Then the run needs a person: it goes to the dead-letter list described on failures, with the step that could not be undone.
What a checkpoint costs, and how often to write oneGranularity is a trade between rework and write load
A checkpoint after every step is a write per step per run. With three lookups inside step 2, the example checkpoints once after the fan-in, not after each lookup. That means a crash in the middle of step 2 repeats all three lookups. Checkpointing each lookup would save that rework and triple the writes. The right granularity follows from the cost of repeating: cheap, side-effect-free reads can share one checkpoint; anything slow, expensive or effectful gets its own.
Checkpoint size matters as much as frequency. A step output that holds a large retrieved document makes every later checkpoint large. Store such outputs once, by reference, and keep the checkpoint to ids and small values. The evidence_ids of the quote workflow are that reference.
In one breath#
A durable run writes down where it stands after every step, so a new process can continue instead of starting over. There is always a gap between doing a step and recording it, so any step that may run twice must be safe to run twice, which for a write means a stable idempotency key checked by the receiver. Retries count attempts in the run state. When a run must stop on purpose, compensations undo the business effects of finished steps in reverse order, which is what a saga is.
What comes next#
You can now keep a run alive across crashes and stop it cleanly. What you cannot yet do is stop a run that is alive and well but should not continue: an agent loop that keeps calling tools, a run that spends without limit, an agent that calls a tool it was never meant to have. The next page, limits, gives each agent its permissions and each run a budget, and makes sure every loop has a reason to end that does not depend on the model.