All chapters
  1. Start hereSomething has to decide what runs next
  2. Foundations · 1Sequence, branch and loop: the three primitives of every workflow
  3. Foundations · 2State machines, DAGs, events and agent loops: four shapes of control
  4. Patterns · 3One agent with tools, or several specialized agents
  5. Patterns · 4Supervisor, planner-executor, router, swarm and peers: who talks to whom
  6. Patterns · 5Control plane and data plane: where code decides and where a model may
  7. Mechanics · 6Task queues, fan-out and fan-in, concurrency and rate limits
  8. Mechanics · 7Handoffs: what one agent must give the next
  9. Mechanics · 8Human approvals: pause, wait for a person, resume safely
  10. Production · 9Durable workflows: checkpoints, retries, compensation and cancellation
  11. Production · 10Permissions, budgets and termination: what a run may do, spend, and when it must stop
  12. Production · 11Failure propagation, circuit breakers and fallback routes
  13. Production · 12Trace propagation and outcome evaluation across a workflow
  14. Reference · 13Frameworks: LangGraph, Temporal, AutoGen, CrewAI and the provider SDKs
  15. Reference · 14Testing orchestration: transition coverage and fault injection
  16. Reference · 15Advanced: speculative execution, deadlocks, fairness and evaluating many agents
  17. Reference · 16The insurance lab: one quote workflow, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Production · 12

Trace propagation and outcome evaluation across a workflow

A run crosses agents, queues and a human pause. How one trace id follows it through every boundary, and how to judge the outcome of the whole run, not of one step.

What you will understand
Four identifiers
submission_id, run_id, attempt_number, trace_id: what each names, and why mixing them up breaks an investigation.
One trace per run
How the trace id crosses a handoff and a pause of three hours, and what happens when it does not.
Judging the run
Why the outcome of a whole run is judged on its end state, not on any one step.

Three weeks after a quote, the senior underwriter who approved UW-004 asks a simple question. The application said the roof was 9 years old; the provider said 22. The recommendation was escalate, citing R-ROOF-03, and she approved after reading the inspection report. Now an auditor wants to see how that recommendation came about. Which lookups ran? Did any fail? Which passages did the underwriting agent see? How long did the case wait for her?

An engineer opens the logs. There are lines for submission SUB-1004, many of them. Some belong to the run she approved. Some belong to an earlier run for the same submission that failed when the provider was down. The roof lookup appears four times: two attempts in the failed run, one in the successful run, one from a retry of the earlier run that nobody remembers. The assessment agent logged under one id, the underwriting agent under another. The three hours of waiting left no lines at all, and the resume after it started a new request id.

Every piece of evidence is somewhere. None of it is connected. The question "what happened in the run that produced this recommendation?" has no direct answer.

This page is about the one thread that ties a run together across agents, queues and a human pause, and about judging what the run achieved. One note before any output: the agents here are scripted stand-ins and time is simulated, so these results show how the orchestration works, not how good or how fast any model is.

1. Four identifiers that must stay apart#

The confusion in the opening comes from four different things that each get called "the id". On this site they have fixed names and fixed meanings.

IdentifierExampleWhat it namesLives how long
submission_idSUB-1004The business record: one broker's quote requestAs long as the business keeps it; survives across runs
run_idrun-b031One execution of the quote workflow for that submissionOne run; a second run for the same submission gets a new one
attempt_number1, 2One try of one step inside one runOne step; starts at 1 for each step
trace_id32 hex charactersThe record of what one run didOne run, including every retry, handoff and pause

A run id names one execution of the workflow. Two runs for SUB-1004 (a failed one and a successful one) share the submission_id and have different run_id values. An attempt_number belongs to a step: in UW-006 the roof lookup has attempts 1 and 2, inside the same run. A retry never gets a new run_id.

That leaves the record itself. A span is one timed piece of work with a name, a start, an end, a status and attributes: one step, one model call, one tool call, one wait.

A trace is the tree of spans that one run produced, connected by parent links.

The trace id is the identifier every span of that trace carries. On this site the rule is one trace per run. A retry is a new span with its attempt_number, in the same trace. A resume after a suspension continues the same trace id.

One more identifier connects the trace to the business record: an evidence id, such as EV-ROOF-0412, names one piece of evidence gathered in step 2. The recommendation cites evidence by id, the handoff package carries the ids (see handoffs), and the span of the lookup that produced it carries the same id. That is how the auditor gets from "this recommendation" to "this provider answer".

What this is not. This page decides which identifiers a run carries and where the trace must not break. What a trace and a span are in depth, how they are stored and queried, logs, metrics and audit events belong to Observability Engineering.

2. A run's trace#

Here is the trace of UW-006, the case where the roof provider times out twice. The script prints the span tree. It ran in a scratch copy of the example.

from run import load_cases, run_case

r = run_case(load_cases("UW-006")[0], "dag")
print(f"run_id {r['run_id']} · trace_id {r['trace_id']} · {len(r['spans'])} spans")
by_id = {s["span_id"]: s for s in r["spans"]}
for s in r["spans"]:
    depth, p = 0, s["parent_span_id"]
    while p:
        depth, p = depth + 1, by_id[p]["parent_span_id"]
    attempt = s["attributes"].get("attempt_number", "")
    print(f"{'  ' * depth}{s['name']:<26} {s['kind']:<6} {s['status']:<8} attempt {attempt}")
run_id run-7641 · trace_id 277e5cb91254d144ec1d59fc7539f2b2 · 18 spans
quote_workflow             run    ok       attempt 
  intake                     step   ok       attempt 1
    intake_agent               model  ok       attempt 
  gather_evidence            step   ok       attempt 1
    assessment_agent           model  ok       attempt 
    get_roof_assessment        tool   error    attempt 1
    search_guidance            tool   ok       attempt 1
    find_similar_decisions     tool   ok       attempt 1
    get_roof_assessment        tool   error    attempt 2
  check_eligibility          step   ok       attempt 1
    check_eligibility          rules  ok       attempt 
  draft_recommendation       step   ok       attempt 1
    underwriting_agent         model  ok       attempt 
  verify_process             step   ok       attempt 1
    verify_process             guard  ok       attempt 
  record_recommendation      step   ok       attempt 1
    record_recommendation      tool   ok       attempt 
  await_underwriter          human  waiting  attempt 1

Read it as the answer to the auditor's questions. One trace_id holds everything this run did. The two roof attempts are two spans with attempt_number 1 and 2, both error, under gather_evidence, next to the two lookups that succeeded. Three different agents (intake, assessment, underwriting) appear as model spans in the same tree. The last span is the wait for the underwriter, still open: its status is waiting.

The trace and span ids in the example follow the shape of the W3C Trace Context recommendation: a 32-hex-character trace id and a 16-hex-character id for each span. That recommendation defines how the ids cross a process boundary in HTTP headers (traceparent carries the version, the trace id, the parent span id and flags). The example is a single process and passes the ids in memory; the shape is borrowed so that the ids would fit a real tracing system unchanged.

Diagram: One run, one trace · across two handoffs and a three-hour pause

Read left to right. The orchestrator creates the run id and the trace id when the run starts. Every step, agent call and lookup is a span in that trace. At each handoff the trace context travels in the package; at the pause it is saved in the checkpoint and read back by the resume. The submission id outlives the run. The agents are scripted stand-ins and time is simulated.

  1. 1 · Start
    1. Run starts: The orchestrator creates both identifiers once, for this execution. The submission id comes from intake.
  2. 2 · Agents and lookups
    1. Handoff: The assessment agent's spans become children in the same trace because the trace context was passed on.
    2. Lookup spans: Each tool call is a span. A retry is another span with attempt_number 2, in the same trace.
  3. 3 · The pause
    1. Suspend: No process holds the run for three hours. The trace id is saved with the run state.
    2. Resume: The senior underwriter's decision continues the stored trace and adds a resume span.
  4. 4 · Judge the run
    1. Outcome check: Compares route, recommendation, citations, missing, approver role and status with the case's expected block.

3. Trace propagation: where the thread breaks#

Inside one function call, keeping the trace id is automatic: the child span is created with the parent's id. The thread breaks at boundaries, and a workflow is mostly boundaries.

Trace propagation means passing the trace context (the trace id and the current span id) across a boundary, so the work on the other side joins the same trace as a child instead of starting a new one. The quote workflow has three kinds of boundary:

BoundaryWhat must carry the trace context
A handoff from one agent to anotherThe handoff package or its envelope
A queue between the orchestrator and a workerThe queue item
A suspension and resumeThe checkpoint; the resume reads the trace id back from it

The pause is the easy one to forget. When the run suspends, no process holds it. The resume three hours later is a new request, often handled by a different process. If the trace id is not in the checkpoint, the resume starts a new trace, and the underwriter's decision ends up in a record that has no visible connection to the evidence it decided on.

The example can switch propagation off at a handoff or at the resume (Engine.break_trace). The script runs UW-004, the case with two handoffs and a three-hour wait, three ways:

from run import load_cases, run_case

case = load_cases("UW-004")[0]           # escalate; a senior underwriter approves after a wait
for handoff, resume in ((True, True), (False, True), (True, False)):
    r = run_case(case, "dag", propagate={"handoff": handoff, "resume": resume})
    roots = [s["name"] for s in r["spans"] if s["parent_span_id"] is None]
    print(f"propagate handoff={handoff!s:5} resume={resume!s:5} -> {len(r['trace_ids'])} trace(s); spans without a parent: {roots}")
propagate handoff=True  resume=True  -> 1 trace(s); spans without a parent: ['quote_workflow']
propagate handoff=False resume=True  -> 3 trace(s); spans without a parent: ['quote_workflow', 'gather_evidence', 'draft_recommendation']
propagate handoff=True  resume=False -> 2 trace(s); spans without a parent: ['quote_workflow', 'resume']

With propagation everywhere, one trace with one root. Lose it at the handoffs, and the run becomes three traces: the evidence gathering and the drafting each start a new tree with no parent, an orphan. Lose it at the resume, and the senior underwriter's decision sits in a second trace of its own. The work was identical in all three runs. Only the record differs, and the auditor's question is answerable only in the first.

For a backend engineer. Spring applications get trace propagation over HTTP and messaging from their tracing library: the headers are added on the way out and read on the way in. The parts of a workflow that are not requests (a checkpoint written today and read in three hours, a scheduler that fires the approval timeout) carry no headers. There the trace context must be stored as data, next to the run state, and restored on purpose.
The same idea in a coding agent. A coding agent's work on one issue spans a planning session, several edit-and-test turns, a pull request, a CI run and a review that may come a day later. If each has its own id, "why did this change get merged?" means joining four systems by hand. Carrying one run id from the issue through the commit messages and the pull request into CI is the same discipline. CI and review are covered on Diff to Deploy.

4. Try it: break the thread#

The lab runs UW-004 with two switches: propagate at the handoffs, propagate at the resume. It draws the spans of the run grouped by trace, and marks each orphan.

The same information without JavaScript, from the run above:

Propagate at handoffsPropagate at resumeTracesSpans without a parent
yesyes1quote_workflow (the root)
noyes3quote_workflow, gather_evidence, draft_recommendation
yesno2quote_workflow, resume

5. Outcome evaluation: judging the whole run#

A connected trace tells you what happened. It does not tell you whether the run did its job. Every span of UW-006 can be ok except the two roof attempts, and the run can still have done exactly the right thing: retried, taken the fallback route, and recommended request_info. Another run can have every span ok and still be wrong: UW-008 in loop mode skipped the guidance search without any span failing.

Outcome evaluation judges a run by its end state against what the case requires: the route, the recommendation, the citations, what is listed as missing, which role must decide, and the final run status. In the example it is run.py:check_expected, which compares a run with the expected block of its case file and returns the differences. An empty list means the run did what the case requires. It is the line expected: as stated in the case at the end of every run printed on this site.

Why the end state, and not each step? Because in loop mode the steps are allowed to vary. UW-010 runs the input of UW-004 twenty times in loop mode: the stand-in model takes nine different paths, and the route is escalate in all twenty. In one trial the path skipped search_guidance, the guard blocked the write, and the run failed, which is the correct outcome for that path. Judging the path would call nineteen good runs "different"; judging the outcome finds the one that failed, and the trace tells you why.

A good outcome check includes the process facts that the outcome depends on, which is where the trace comes back in. "Recommendation decline, citing R-ROOF-01" is the right outcome for UW-008 only if the guidance search ran; that fact is in the trace, and the guard checked it at run time. Outcome and trace are two halves of one judgment.

QuestionAnswered by
Did the run end in the state the case requires?The outcome check (check_expected)
Did the required process happen (eligibility ran, cited rules were retrieved)?The guard at run time; the trace afterwards
Why did this run take this path?The trace
How often does the workflow produce the right outcome across many trials?An evaluation over many runs: Eval Engineering
What this is not. One outcome check on one run is a test, not a measurement. Measuring how often a workflow succeeds across many trials, with graders, golden sets and release gates, is the subject of Eval Engineering.

Who decides here#

DecisionWho
Which ids a run carries and where they are createdCode: the orchestrator creates run_id and trace_id at the start of a run
Where trace context is passed onCode: at every handoff, queue item and checkpoint
What counts as the right outcome of a runPeople, in the case's expected block, versioned with the cases
Whether a run met itCode: check_expected
What a model decides hereNothing about the record. Its calls appear as spans like any other step

Checklist#

One trace per run, or one per attempt?The choice this site made, and the other one

This site uses one trace per run and records retries as spans with an attempt_number. The alternative, a new trace for each attempt of the whole run (common when each attempt is a separate request), keeps traces small and makes "how often does the first attempt fail?" easy to count. It makes "what happened to this submission?" harder, because the answer is spread over traces that must be joined by run_id.

The choice matters most for long runs. A run that waits three days for an underwriter has a trace that is open for three days. Some tracing systems handle long traces poorly. A common compromise is a trace per active segment (before the pause, after the resume) with an explicit link between them, carrying the same run_id. That is not an orphan: the link is recorded on purpose. What matters is that the decision is written down and that every segment can be found from the run.

What a span may carryPrompts, passages, personal data and the audit trail

A span of a model call can carry the prompt and the output. That is invaluable for debugging and dangerous for privacy: a real submission contains names, addresses and financial details, and traces are often kept longer and seen by more people than the business record. The usual rule is to put ids in spans (submission_id, evidence ids, rule ids) and keep content in the systems that own it, with their own access control. The example's spans follow that rule: they carry ids and outcomes, not the email.

An audit trail is not a trace. A trace may be sampled, dropped or expired; an audit record of who decided what must not be. In the quote workflow the human_decision lives in the store, and the trace points to it. How audit events are kept is the subject of Observability Engineering.

Outcome evaluation when several agents share the resultCredit, blame and the counterfactual run

When the outcome is wrong, the trace shows which agent did what, but not which agent caused the error. The assessment agent may have handed over a correct package that the underwriting agent misread, or a package that was missing a field the underwriting agent could not have known to ask for. A useful technique is the counterfactual run: replay the run with one agent's output replaced by the correct one (from the case file) and see whether the outcome becomes right. With scripted stand-ins and recorded tool results this is cheap; with live models it needs the recorded results that durable workflows keep anyway. The advanced page discusses evaluating multi-agent systems further.

In one breath#

Keep four identifiers apart: the submission is the business record, the run is one execution, the attempt is one try of one step, and the trace is the record of the run. One trace per run, carried across every handoff, queue and pause, with retries as spans, so the question "what happened in the run that produced this recommendation?" has one answer. Then judge the run by its end state against what the case requires, and use the trace to explain it.

What comes next#

You now have the concepts: shapes, topologies, the control plane, concurrency, handoffs, approvals, durability, limits, failures and tracing. Each of them exists in some form in the frameworks people use to build agent workflows. The next page, frameworks, reads LangGraph, Temporal, AutoGen, CrewAI and three provider SDKs on their own documentation and maps each concept onto what they run for you and what stays your job.