All chapters
  1. Start hereSomething has to decide what runs next
  2. Foundations · 1Sequence, branch and loop: the three primitives of every workflow
  3. Foundations · 2State machines, DAGs, events and agent loops: four shapes of control
  4. Patterns · 3One agent with tools, or several specialized agents
  5. Patterns · 4Supervisor, planner-executor, router, swarm and peers: who talks to whom
  6. Patterns · 5Control plane and data plane: where code decides and where a model may
  7. Mechanics · 6Task queues, fan-out and fan-in, concurrency and rate limits
  8. Mechanics · 7Handoffs: what one agent must give the next
  9. Mechanics · 8Human approvals: pause, wait for a person, resume safely
  10. Production · 9Durable workflows: checkpoints, retries, compensation and cancellation
  11. Production · 10Permissions, budgets and termination: what a run may do, spend, and when it must stop
  12. Production · 11Failure propagation, circuit breakers and fallback routes
  13. Production · 12Trace propagation and outcome evaluation across a workflow
  14. Reference · 13Frameworks: LangGraph, Temporal, AutoGen, CrewAI and the provider SDKs
  15. Reference · 14Testing orchestration: transition coverage and fault injection
  16. Reference · 15Advanced: speculative execution, deadlocks, fairness and evaluating many agents
  17. Reference · 16The insurance lab: one quote workflow, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Mechanics · 6

Task queues, fan-out and fan-in, concurrency and rate limits

Three lookups can run at the same time. Queues and workers, limits on parallel work, and what to do when two branches return and the third does not.

What you will understand
Fan-out, fan-in
How three lookups run at the same time, and how their answers become one input.
Two kinds of limit
Why "how many at once" and "how many per second" are different limits that protect different things.
The third answer
What to do when two branches return and the third does not: the aggregation policy.

Step 2 of the quote workflow, gather_evidence, needs three things: the roof assessment from the roof-data provider, the passages of the guidance that apply, and similar past decisions. None of them depends on another. Done one after the other, with an ordinary day's timings in the example (180, 240 and 150 simulated milliseconds), step 2 takes 570 ms. Done at the same time, it takes as long as the slowest, 240 ms.

That looks like free speed. Then Monday comes. The roof provider times out on the first try (2,000 ms) and answers on the second. The guidance search is slow (1,500 ms). Two hundred quote requests arrive in the same hour, each starting three lookups. The provider's contract allows a fixed number of calls per second. The orchestrator has a fixed number of workers.

Now four questions appear that a single lookup never raised. How many lookups may run at once? How many may start per second? How long does the step wait for a slow branch? And when two answers are in and the third is not, does the step continue, wait, or give up?

This page answers them in that order. One note before any output: the agents here are scripted stand-ins and time is simulated, so these results show how the orchestration works, not how good or how fast any model is.

1. Fan-out and fan-in#

Start with the shape. Fan-out means starting several independent pieces of work from one point, so they can run at the same time. Fan-in means waiting for those pieces and combining their results into one input for the next step. In the quote workflow, step 2 fans out to roof_assessment, guidance and similar_decisions, and fans in before step 3, check_eligibility.

A tiny example in plain terms: an underwriter's assistant sends three emails at once (to the roof surveyor, to the guidance team, to the archive), waits for the replies, and staples them into one folder. Sending them one at a time would only make the folder later.

For a backend engineer. This is CompletableFuture.allOf(roof, guidance, similar) followed by a thenApply that reads all three. The difference that matters: allOf completes exceptionally if any future fails. The quote workflow does not want that. One missing optional answer should not throw away two good ones, so the fan-in here is a rule, not allOf.

2. Queues and workers#

Something has to run the lookups. A worker is the thing that executes one piece of work at a time: a thread, a process, a container. A task queue is the list of pieces of work waiting for a worker. The orchestrator puts work on the queue; a free worker takes the next item.

The example simulates exactly that in core.py:simulate_fanout. Each lookup attempt is a queue item with the time it becomes ready. A free worker takes the earliest one. A failed attempt goes back on the queue, ready after a backoff of 500 ms, with its attempt_number raised by one. Time is simulated, so nothing sleeps.

The number of workers is the concurrency limit: the most pieces of work that may run at the same moment. It protects the thing doing the work (your workers, memory, connections) and the dependency (a provider that slows down when hit by many calls at once).

The script below runs Monday's three lookups twice: with three workers and with one. It ran in a scratch copy of the example.

import core

outcomes = {"roof_assessment": ["timeout", "ok"], "guidance": ["slow"], "similar_decisions": ["ok"]}
for limit in (3, 1):
    fan = core.simulate_fanout(outcomes, concurrency_limit=limit, deadline_ms=5000)
    print(f"concurrency limit {limit}: finished at {fan['finished_ms']} ms · {fan['status']} · {fan['decision']}")
    for a in fan["timeline"]:
        print(f"  worker {a['worker']}  {a['start_ms']:>5}-{a['end_ms']:<5} ms  {a['branch']:18} attempt {a['attempt']}  {a['outcome']}")
concurrency limit 3: finished at 2680 ms · {'roof_assessment': 'ok', 'guidance': 'ok', 'similar_decisions': 'ok'} · continue
  worker 0      0-2000  ms  roof_assessment    attempt 1  timeout
  worker 1      0-1500  ms  guidance           attempt 1  slow
  worker 2      0-150   ms  similar_decisions  attempt 1  ok
  worker 2   2500-2680  ms  roof_assessment    attempt 2  ok
concurrency limit 1: finished at 3830 ms · {'roof_assessment': 'ok', 'guidance': 'ok', 'similar_decisions': 'ok'} · continue
  worker 0      0-2000  ms  roof_assessment    attempt 1  timeout
  worker 0   2000-3500  ms  guidance           attempt 1  slow
  worker 0   3500-3650  ms  similar_decisions  attempt 1  ok
  worker 0   3650-3830  ms  roof_assessment    attempt 2  ok

Read the first timeline. The retry of the roof lookup ran on worker 2, which had been free since 150 ms: the queue gave it to whichever worker was free. With one worker, every lookup waits for the one before it, and the step ends 1,150 ms later. All three answers arrive in both cases. What this shows: the concurrency limit changes when results arrive, not what they are. What it does not show: real network behavior. The durations are illustrative constants in core.py.

Diagram: Step 2 as fan-out and fan-in · three workers, one retry, one rule

Read left to right. The orchestrator puts three lookups on the queue. Three workers take them at once. The roof lookup times out and goes back on the queue after a 500 ms backoff; the first free worker takes the retry. The fan-in rule combines what arrived before the deadline. Durations are illustrative; time is simulated.

  1. 1 · Fan-out
    1. Task queue: Each lookup attempt waits here until it is ready and a worker is free.
  2. 2 · Workers (limit 3)
    1. Worker 0: Runs the first roof attempt; it times out.
    2. Worker 1: Runs the guidance search; it is slow but answers.
    3. Worker 2: Finishes early, then takes the roof retry when it becomes ready.
  3. 3 · Fan-in
    1. Fan-in rule: Combines what arrived before the 5000 ms deadline. Missing required: request_info. Missing optional: reported.
  4. 4 · Next step
    1. check_eligibility: The rules read one record built from the three answers, never the raw calls.

3. The deadline: how long the step may wait#

A step that waits for its slowest branch inherits that branch's worst day. A deadline for the whole fan-out bounds this: results that arrive after it are not used, and attempts that cannot start before it are not started. The example's default is 5,000 simulated ms (illustrative).

The deadline and the concurrency limit interact. Here is the same Monday with a tighter deadline of 3,000 ms:

import core

outcomes = {"roof_assessment": ["timeout", "ok"], "guidance": ["slow"], "similar_decisions": ["ok"]}
for limit in (3, 1):
    fan = core.simulate_fanout(outcomes, concurrency_limit=limit, deadline_ms=3000)
    print(f"limit {limit}, deadline 3000 ms: {fan['status']} -> {fan['decision']} {fan['missing_required']}")
limit 3, deadline 3000 ms: {'roof_assessment': 'ok', 'guidance': 'ok', 'similar_decisions': 'ok'} -> continue []
limit 1, deadline 3000 ms: {'roof_assessment': 'failed', 'guidance': 'late', 'similar_decisions': 'late'} -> request_info ['roof_assessment', 'guidance']

With three workers everything fits. With one worker, the guidance search finishes at 3,500 ms, after the deadline; the similar-decisions lookup cannot start before it; the roof retry never runs. Nothing was wrong with any provider in the second run. The run asks the broker for information because of how the orchestrator queued the work. Limits are part of the result, and the record says which branches were late, so the cause can be found.

4. The aggregation policy: when is enough enough?#

Now the fourth question. Two answers are in and the third is missing. An aggregation policy is the rule that decides what the fan-in does with what arrived: which branches are required, which are optional, and what happens when a required one is missing.

The quote workflow's default rule: roof_assessment and guidance are required, similar_decisions is optional. Without a roof assessment the rules cannot be applied; without the guidance the guard cannot check the cited rule. Similar past decisions are context for the underwriter, not an input to any rule. A missing required branch gives request_info, with the branch named in missing. The rule is five lines:

def fan_in(branch_status: dict[str, str], rule: str = "default") -> dict[str, Any]:
    """Aggregate the lookups. A missing required branch gives request_info; a missing optional one is reported."""
    required = FAN_IN_RULES[rule]
    arrived = [b for b in BRANCHES if branch_status.get(b) == "ok"]
    missing_required = [b for b in BRANCHES if b in required and b not in arrived]
    missing_optional = [b for b in BRANCHES if b not in required and b not in arrived]
    return {"aggregated": arrived, "missing_required": missing_required, "missing_optional": missing_optional,
            "decision": "request_info" if missing_required else "continue"}

That is core.py:fan_in. The example has three rules in FAN_IN_RULES: default, all (every branch required) and none (nothing required). With similar_decisions failed, default and none continue and report it missing; all sends the run to request_info. The case UW-903 is the default rule at work: similar_decisions fails twice, guidance is slow, and the run continues to a decline with the gap recorded.

$ $PY run.py --case UW-903
Quote workflow · scripted stand-in agents (no model called) · simulated time · seed 7
UW-903 · One lookup fails and one is slow · dag mode · run-602c · trace 21ca3b4648ab...
  path:      intake > gather_evidence > check_eligibility > draft_recommendation > verify_process > record_recommendation > await_underwriter
  attempts:  find_similar_decisions attempt_number 2 error
  fan-in:    roof_assessment ok [ok], guidance ok [slow], similar_decisions failed [error, error] · missing similar_decisions
  guard:     pass (Eligibility ran and every cited rule was retrieved.)
  recommendation: decline · cites R-ROOF-01 · missing - · policy 2026.2 · stored row status stored
  human_decision: none yet (waiting for: underwriter)
  status:    suspended · model calls 3 · tool calls 5 · simulated time 4.3 s
  expected:  as stated in the case
wrote 1 run file(s) in out/runs/ and out/summary.json

The model recommends; a person decides, and the underwriter sees that the similar-decisions lookup is missing. The aggregation policy is the place where the meaning of each branch is written down. "Required" is a statement about the business, not about the provider: no rule may run without a roof age.

Common confusion. "Wait for all" and "wait for the first" are not the only choices. The quote workflow waits for all required branches, accepts the optional one if it is in time, and has a deadline for everything. Other workflows want a quorum (two of three sources agree) or the first good answer from several equivalent providers. Each is a different aggregation policy, and each must say what it does with a late answer.
Who decides here. Code decides everything on this page: the number of workers, the deadline, the retry, and the fan-in rule. In pipeline mode a model decides nothing about step 2. In loop mode a model may choose the order of the lookups inside its budget (see control plane), but it still cannot call more of them at once than the workers allow, and the guard after the loop checks the same required evidence.

5. Rate limits and backpressure#

The concurrency limit counts work in progress. A rate limit counts work started per unit of time: for example, at most 10 calls per second to the roof provider (an illustrative number). They protect different things. A provider can be fine with 50 slow calls in flight and still refuse the 11th call that starts within one second; a provider can allow 100 calls per second and still collapse when each one holds a connection for a minute.

In the quote workflow a rate limit belongs to a dependency, not to a run. Two hundred runs each starting one roof lookup share the provider's allowance. That is why it lives in a shared component in front of the provider, often next to the circuit breaker. The example does not implement a rate limit; this section describes the mechanism, not code that ran.

When work arrives faster than it can be done, the queue grows. Backpressure is any mechanism by which a busy downstream part makes the upstream part slow down, instead of letting the queue grow without limit. In the example the only backpressure is the simplest one: an attempt waits in the queue until a worker is free. In production the choices are to reject new runs (with a clear answer to the broker's system), to delay starting new runs, or to shed optional work first, for example by not calling similar_decisions while the queue is long. The last option is possible here because the aggregation policy already says that branch is optional.

For a backend engineer. A Kafka consumer group gives you a concurrency limit for free: one consumer per partition, so the number of partitions caps the parallel work, and lag is the visible queue. It does not give you a rate limit toward a third-party API; that is still yours. The comparison stops at fan-in: Kafka has no built-in "wait for these three messages and combine them", which is exactly what an orchestrator adds.
What this is not. This page decides how the quote workflow schedules its own work. Operational guarantees for the whole system under load (load shedding policy, service levels, delivery semantics of a queue) belong to Agent Reliability Engineering. How a lookup tool is defined and executed belongs to Tools Engineering.
The same idea in a coding agent. When a coding agent opens a pull request, CI fans out: lint, unit tests, integration tests, a security scan, all at once on separate runners. The fan-in is the required status checks: the merge waits for the required ones, and an optional job may fail without blocking. Runner pools are the concurrency limit. CI and its gates are covered on Diff to Deploy.

6. Try it: schedule the three lookups#

The lab sets the outcome of each lookup (ok, slow, timeout, error), the concurrency limit, the deadline and the fan-in rule. It draws the timeline per worker, shows what was aggregated and what is missing, and the route the run takes.

The same information without JavaScript, from the runs above:

LookupsWorkersDeadlineFan-in ruleResult
roof timeout then ok, guidance slow, similar ok35,000 msdefaultall arrive at 2,680 ms; continue
same15,000 msdefaultall arrive at 3,830 ms; continue
same33,000 msdefaultall in time; continue
same13,000 msdefaultroof failed, guidance late; request_info
roof ok, guidance slow, similar error twice (UW-903)35,000 msdefaultsimilar missing (optional); continue to decline
same, similar failed35,000 msallsimilar missing (required); request_info

Checklist#

Head-of-line blocking and priorityWhy a retry should not wait behind new work, or jump ahead of it

In the example a retry joins the same queue as first attempts, ordered by when it becomes ready. That is fair and simple, and it has a cost: a run that is retrying its roof lookup can wait behind the first attempts of many new runs, so the oldest run finishes last. The opposite choice (retries first) lets one broken provider fill the workers with retries. Production queues usually separate the two: a priority by run age, with a cap on the share of workers that retries may use.

Head-of-line blocking also happens across branches. If all workers are busy with slow guidance searches, a fast roof lookup waits. Separate pools per dependency avoid that, at the cost of idle workers in one pool while another is full.

What to do with a late resultDiscard, record, or use next time

After the deadline the fan-in has decided. A result that arrives later cannot change this run's route without breaking the record of why the route was taken. The example discards it for the route and marks the branch late. Two production choices are worth knowing. Record it anyway, with a flag, so the underwriter can see it in the cockpit next to the request_info recommendation. Or cache it, keyed by the submission, so that the next run for the same submission (after the broker answers) does not call the provider again. Both are fine; silently feeding a late result into a step that already ran is not.

A late result also costs: the provider did the work and the call was paid for. A deadline that is often missed is a signal to change the concurrency limit, the provider timeout, or the provider.

Fanning out to agents, not lookupsParallel model calls and the aggregation problem

The three lookups here are tools: deterministic answers, easy to combine. Fanning out to several agents (for example three agents that each assess one risk and write a paragraph) moves the hard part into the fan-in. Paragraphs do not merge like fields. Someone has to reconcile them: a code rule over structured outputs, a further model call, or a person. Each choice needs its own guard and its own evaluation.

A common pattern is to fan out to agents that return structured records (the shape is the subject of Structured Output Engineering), and to keep the fan-in in code. The topologies page shows who decides in such arrangements, and advanced discusses evaluating them.

In one breath#

Fan-out starts independent work at the same time; fan-in combines it with an aggregation policy that says which branches are required and what a missing one means. Workers take work from a queue, and the concurrency limit caps how much runs at once, while a rate limit caps how much starts per second toward a shared dependency. A deadline bounds the wait, and limits are part of the result: with one worker instead of three, the same healthy providers can turn a run into request_info.

What comes next#

Step 2 combined three answers inside one step. The next problem is larger: the work moves from one agent to another. The assessment agent has gathered evidence; the underwriting agent must now write the recommendation. What goes across, what stays behind, and who is responsible once it has gone? The next page is handoffs.