Your Sub-Agents Are a Distributed System. Treat Them Like One. - Girish Konda, Microsoft
Treat every sub-agent handoff like a network call: enforce explicit outcomes, freshness checks, bounded fanout, and safe handling of side effects.
- Return named states such as completed, failed, stale, or unknown—not just free-form text.
- Attach source, version, and timestamps to facts; check freshness when using them.
- Give each side effect a unique identity, require approval for that action, and read back its outcome instead of blindly retrying.
The practical examples show how to enforce safe handoffs in systems that perform real writes. · Best for: Engineers building orchestrated agents that use tools or affect production systems.
Girish Konda argues that concurrent sub-agents sharing state and calling real tools form a distributed system. He proposes a “safe handoff” model: source-backed facts, at-most-once effects, bounded fanout, and explicit outcomes enforced by the orchestrator.
Sub-agents inherit distributed-system failures0:00
- Workers run concurrently, share mutable context, call external systems, and return results for an orchestrator to merge.
- Expect partial failures, dirty reads, unbounded fanout, and retries that may repeat side effects.
- A worker can return fluent, success-shaped text despite doing only part of the work.
Timeouts and outcomes must be explicit2:48
- A timeout cannot tell you whether a worker is still running, crashed, or whose reply was lost; treat it as unknown, not proof of failure.
- Represent outcomes as named states and distinguish a failed check from a successful check that returned “no.”
- A worker’s text may claim completion even when its protocol status does not.
Shared context needs provenance and freshness8:18
- Workers repeating the same unsupported claim are not independent confirmations if they all read it from shared context.
- Record each fact’s claim, source, version, scope, and relevant times; check freshness when a decision uses it.
- Separate observations backed by a source and read time from inferences.
Make side effects safe to retry12:37
- A lost reply leaves a write’s outcome uncertain; retrying the task may duplicate the side effect.
- Give each side effect an identity, bind approval to the exact action, use it once, and read back the effect by identity.
- Treat submitted as unconfirmed; report confirmed only after observing the result.
Bound fanout and make refusals visible15:35
- Use per-downstream quotas, bounded concurrency, and wait limits; expose refusal as a failure the orchestrator can handle.
- Reserve capacity before starting synchronous work to avoid overbooking.
- Enforce checks in orchestrator code: validate schema, evidence, freshness, and recomputed values; refuse invalid results.
Limits and when the model fits19:31
- Safe handoff cannot make source facts true, guarantee exactly-once delivery, roll back partial work, or impose a global order across workers.
- It adds reads, latency, and refusals; it is likely overhead for drafts a person reviews before action.
- The model assumes a single orchestrator; peer agents without one need another approach.
Step by step
- Represent every handoff with a named outcome, including partial, stale, unknown, failed, or rejected.
- For each fact, record its source, version, scope, and timestamps; reject it if it exceeds the freshness limit at decision time.
- Before a side effect, assign it an identity and require approval bound to that exact action; do not blindly replay an uncertain call.
- Read back side effects by identity and mark them confirmed only when observed.
- Set concurrency and wait budgets per downstream, reserve capacity before starting work, and return visible refusals.
- Have the orchestrator validate result schema, evidence, freshness, and calculations in code; refuse unsupported claims.
Tools & resources mentioned
- MCP — tool protocol discussed for representing tool results and errors.
- MCP Peacemaker — the speaker’s bridge project, used for examples involving timeouts, side effects, and fanout limits.
- Lamport’s 1978 work — cited for ordering events and recording fact provenance.
- FLP paper (1985) and Chandra–Toueg (1996) — cited for limits of agreement and failure detectors in asynchronous systems.
- Martin Kleppmann’s distributed-locking post — cited for leases and fencing tokens.
Notes are AI-generated from the video's transcript and may miss nuance.