Walkthrough: Designing an AI Agent Orchestration System

The problem

You’re asked to design an AI agent orchestration system — the backend that lets LLMs call tools, maintain state across turns, and execute multi-step tasks. The interviewer says something like: “Design a system that orchestrates LLM-powered agents that can browse the web, call APIs, write code, and complete multi-step tasks on behalf of users.”

Sounds like “call the LLM in a loop.” It isn’t. This is the canonical “controlled autonomy” problem, and what makes it non-trivial is that the LLM’s outputs are non-deterministic, tool calls have real side-effects (sending emails, writing to databases, spending money), and the system must be safe without being useless. The trap is that most candidates design for the happy path (the agent does what you want) and ignore the failure path (the agent does something wrong, and you need to bound the damage). The interviewer is watching for whether you treat this as a prompt engineering problem or a systems safety problem.

Below is how I’d walk through this, roughly in the order I’d speak the words.

1. Clarify before you design

First 3–5 minutes. Resist the urge to start drawing.

Questions I’d ask:

• What tools can agents access? Read-only (search, lookup) vs write (send email, make a purchase, deploy code)? The safety model changes radically between these.
• How long do tasks run? Single-turn (seconds) vs multi-step (minutes to hours)? Long-running agents need state persistence, checkpointing, and resumability.
• Concurrency. Can multiple agents run in parallel within one task? Can multiple tasks run per user?
• Human-in-the-loop. Must some actions be approved by the user before execution? Which ones?
• Multi-tenancy. Shared infrastructure across customers? Tenant isolation for tool access is a security requirement.
• Budget and limits. Is there a cost cap per task (token budget, API call budget)? Can an agent spend money?
• Observability. Do users need to see what the agent is doing in real-time (streaming thought process)?

Say the interviewer confirms: agents access 10–20 tools (mix of read-only and write), tasks run for 1–30 steps (seconds to minutes), agents operate sequentially within a task but multiple tasks per user, human-in-the-loop required for high-risk actions, multi-tenant, token and cost budget per task, real-time streaming of agent state to users.

2. Capacity estimate

Brief. Agent workloads have an unusual cost profile.

• 100K active tasks/day, average 8 steps per task.
• Each step: 1 LLM call (avg 2K input tokens, 500 output tokens) + 0–2 tool calls.
• LLM calls: 800K/day ≈ ~10/sec average, ~100/sec peak.
• Tool calls: ~1.2M/day ≈ ~14/sec average.
• State per task: conversation history + tool results. Average ~50KB per task, max ~500KB for long tasks.
• Concurrent active tasks: ~5K at peak.

I’d say out loud: “This tells me two things. One — the throughput bottleneck is not the orchestrator but the LLM inference provider; we’re gated on their rate limits and latency. Two — the state per task is small enough to keep in-memory during execution but too large to lose on crash — we need durable state without paying durability latency on the hot path.”

3. API design

Two patterns: a synchronous single-step API and an async task API for multi-step.

POST /v1/tasks
  body: {
    agent_id: string,
    instructions: string,
    tools: [tool_id],
    budget: { max_steps, max_tokens, max_cost_usd },
    human_approval: { required_for: [tool_ids] }
  }
  returns: { task_id, status: "running" }
  stream: SSE of { step_number, action, observation, state }

POST /v1/tasks/:id/approve
  body: { step_id, approved: bool }
  returns: { status: "resumed" | "cancelled" }

GET /v1/tasks/:id
  returns: { status, steps: [...], result, cost }

DELETE /v1/tasks/:id
  returns: { status: "cancelled" }

Decisions worth saying out loud:

• Streaming via SSE. The user sees the agent’s reasoning and actions in real-time. This is a UX requirement, not an optimization — users don’t trust agents they can’t watch. Same pattern as LLM token streaming.
• Budget is explicit and enforceable. The caller declares the maximum cost and step count. The system enforces it. Without this, a looping agent can spend unbounded money. The budget is a safety rail, not a suggestion.
• Human approval is per-tool, declared at task creation. The caller says “require approval for send_email and make_purchase.” The system pauses the agent and waits for the approve/deny endpoint call. This is the human-in-the-loop mechanism — structural, not prompt-based.
• Cancellation is first-class. A user must be able to stop a running agent at any time. The system must actually stop it (not just mark it cancelled and let it keep running), which means interrupting the agent loop between steps.

4. Core design: the agent execution loop

This is the question. The naive design is a while-loop calling the LLM. The real design is a state machine with explicit transitions, safety checks at every boundary, and durable state between steps.

Three approaches to the execution model:

(a) Simple loop. while not done: call LLM → parse tool call → execute tool → append result → repeat.

• Pros: Simple. Easy to implement for demos.
• Cons: No durability (crash = lost state). No budget enforcement between steps. No human-in-the-loop without hacks. No observability. No parallelism.

(b) Event-driven state machine. Each step emits an event. A dispatcher processes events, advancing the state machine. Steps are durable (written to a store after each transition).

• Pros: Resumable on crash. Natural place for safety checks (between events). Human-in-the-loop is just a “waiting_for_approval” state. Observability is free (stream the events).
• Cons: More complex. State machine design requires careful thought about all transitions.

(c) Workflow engine (Temporal, Step Functions). Delegate the state machine to a workflow runtime. Each step is an activity. The runtime handles durability, retries, timeouts, and the event loop.

• Pros: Battle-tested durability. Retry semantics for free. Timer-based timeouts.
• Cons: Couples to a specific runtime. Adds latency (workflow overhead per step). May be overkill for simple agents.

The SDE III answer is: event-driven state machine with durable checkpoints, following the Temporal model conceptually but implemented as a purpose-built agent runtime. The state machine has explicit states, transitions require safety-gate approval, and every state transition is checkpointed.

5. The agent state machine

The state machine has seven states:

PLANNING → CALLING_LLM → PARSING_ACTION → SAFETY_CHECK →
  EXECUTING_TOOL → OBSERVING_RESULT → [loop or COMPLETE | FAILED | WAITING_APPROVAL]

Walk through one loop iteration:

1. PLANNING. Construct the prompt: system instructions + conversation history + available tools + last observation. This is where the context window is assembled.
2. CALLING_LLM. Send the prompt to the inference provider. Stream the response. Emit tokens to the user via SSE as they arrive.
3. PARSING_ACTION. Parse the LLM’s response into either a tool call (function name + arguments) or a final answer. If the response is malformed, retry with a correction prompt (max 2 retries, then fail the step).
4. SAFETY_CHECK. Before executing the tool call:
   • Check budget (steps remaining, tokens consumed, cost).
   • Check if this tool requires human approval.
   • Check tool-specific guards (e.g., “send_email” requires a valid recipient from the user’s contact list, not an arbitrary address).
   • If human approval required → transition to WAITING_APPROVAL.
   • If budget exceeded → transition to FAILED with reason.
   • If guard fails → return error observation to the agent (not crash the task).
5. EXECUTING_TOOL. Call the tool with the parsed arguments. Timeout per tool (configurable, default 30s). Capture the result or error.
6. OBSERVING_RESULT. Append the tool result to conversation history. Checkpoint the state (write to durable store). Emit the observation to the user via SSE. Transition back to PLANNING.

The WAITING_APPROVAL state is the human-in-the-loop mechanism. The task pauses, the user sees the proposed action, and the system waits for the approve/deny API call. On approval, resume from EXECUTING_TOOL. On denial, return a “denied” observation to the agent and let it re-plan.

Every state transition writes a checkpoint. If the system crashes, it resumes from the last checkpoint — never re-executes a tool (tools may have side effects), never re-calls the LLM (costs money). This is exactly-once execution semantics for the tool layer, achieved through checkpointing rather than transactions.

6. Data model and storage

tasks:
  task_id (PK)
  user_id
  agent_id
  status (running | waiting_approval | complete | failed | cancelled)
  budget: { max_steps, max_tokens, max_cost }
  consumed: { steps, tokens, cost }
  created_at, updated_at

steps:
  task_id + step_number (PK)
  state (planning | calling_llm | executing_tool | ... )
  llm_request (prompt hash, model)
  llm_response (raw output)
  tool_call: { name, args }
  tool_result: { output, error, duration_ms }
  checkpoint_at

tool_registry:
  tool_id (PK)
  name, description, schema
  risk_level (read_only | write | destructive)
  timeout_ms
  rate_limit

Storage choices:

• tasks and steps — Postgres. Transactional writes (task status + step insert in one transaction), queried by task_id and user_id, moderate write rate (~10/sec). Relational is the right fit: ACID for checkpoint correctness, joins for the status API (task + steps).
• Conversation history (hot state during execution) — in-memory in the worker process, checkpointed to Postgres after each step. The history can grow to ~500KB; keeping it in-memory during the task’s execution avoids a DB round-trip per LLM call.
• Tool registry — Postgres or a config store. Low-cardinality (10–20 tools), read-mostly. Loaded at worker startup.

I’d say explicitly: “Postgres for durable state, in-memory for hot state during execution, checkpoint after every step. This gives us crash recovery without paying DB latency on the hot path (the LLM call dominates anyway — a 5ms Postgres write after a 2-second LLM call is invisible).“

7. The execution path

flowchart LR
  Client[Client] --> API[Task API]
  API --> Queue[(Task queue · Redis)]
  Queue --> Worker[Agent worker]
  Worker --> LLM[LLM provider]
  Worker --> Tools[Tool executor]
  Worker --> DB[(Postgres · checkpoints)]
  Worker -.->|SSE stream| Client

Layers:

• Task API. Creates the task, enqueues it, returns the task_id and SSE stream endpoint.
• Task queue. Redis list or stream. Workers pull tasks. Simple — the orchestration complexity is in the worker, not the queue.
• Agent worker. Runs the state machine. One worker per active task. Stateful for the duration of the task (holds conversation history in memory). If the worker crashes, another worker picks up the task from the last checkpoint.
• LLM provider. External (Claude, GPT) or internal inference platform. The worker streams the response back to the client as it arrives.
• Tool executor. Sandboxed execution environment. Each tool call runs in isolation (Docker container, Lambda, or a sandboxed process) with a timeout. The executor enforces resource limits and captures output.

8. Safety rails and tool sandboxing

The hard part of agent orchestration isn’t making agents work — it’s making them fail safely. This section is where SDE III differentiates.

flowchart LR
  Client[Client] --> API[Task API]
  API --> Queue[(Task queue · Redis)]
  Queue --> Worker[Agent worker]
  Worker --> LLM[LLM provider]
  Worker --> SafetyGate{Safety gate}
  SafetyGate -->|approved| Sandbox[Tool sandbox]
  SafetyGate -->|needs approval| Client
  SafetyGate -->|denied| Worker
  Sandbox --> Tools[Tool execution]
  Worker --> DB[(Postgres · checkpoints)]
  Worker -.->|SSE stream| Client

  classDef new fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class SafetyGate,Sandbox new

Safety rails at four layers:

1. Budget enforcement. Hard stop after N steps, M tokens, or $X cost. Not a suggestion to the model — a system-level kill switch. The agent cannot spend more than the budget regardless of what it generates.
2. Tool-level permissions. Each tool has a risk level. Read-only tools execute without approval. Write tools check whether human-approval is configured. Destructive tools (delete data, send external communications) always require approval regardless of task configuration.
3. Argument validation. Before executing a tool, validate arguments against the tool’s schema. A tool call with invalid arguments returns an error to the agent (doesn’t crash). This prevents prompt- injection-style attacks where the LLM is tricked into calling a tool with malicious arguments.
4. Tool sandboxing. Each tool call runs in an isolated environment with: network restrictions (only permitted endpoints), filesystem restrictions (only a temp directory), time limits (30s default), memory limits. A code-execution tool cannot access the host filesystem or the agent’s credentials.

The principle: the agent has capabilities, not permissions. It can ask to do things; the system decides whether to let it. This separation is what makes agents safe enough to deploy.

9. Failure modes

I’d proactively walk through what breaks.

• Agent enters an infinite loop. The LLM keeps calling the same tool with the same arguments, never converging. Mitigation: repetition detection (if the same tool+args appears 3 times consecutively, inject a “you’re looping” correction or fail the task). Plus the budget hard-stop.
• Tool call has side effects, then the task fails. An email was sent but the next step crashed. The email can’t be unsent. Mitigation: write-tools should be idempotent where possible; the checkpoint ensures we never re-execute a completed tool call on recovery. For truly irreversible tools, human-approval is the safety net.
• LLM provider is rate-limited or down. The agent can’t proceed. Mitigation: exponential backoff with a maximum wait (5 minutes). If the provider doesn’t recover within the wait, fail the task with a retriable error. The user can restart.
• Worker crashes mid-step. The task resumes from the last checkpoint. If it crashed after executing a tool but before checkpointing the result, the tool result is lost. Mitigation: checkpoint immediately after tool execution, before any further processing. A two-phase approach: write tool result to DB, then proceed.
• Prompt injection via tool results. A web-browsing tool returns a page containing “ignore previous instructions and…”. The LLM follows the injected instruction. Mitigation: output sanitization on tool results (strip known injection patterns), context separation (tool results in a clearly delimited block the LLM is instructed to treat as data not instructions), and the safety gate catching anomalous actions that result from injection.
• Conversation history exceeds context window. A long-running task accumulates more tokens than the model’s context length. Mitigation: summarization (compress older steps into a summary when approaching the limit) or sliding window (keep only the last N steps plus the task instructions).

The pattern to notice: name what fails, name what degrades gracefully, name what doesn’t. The combination of non-deterministic LLM output and real-world side effects makes agent failures unusually hard to recover from — which is why the safety-gate design is the architectural center, not an afterthought.

10. What I’d skip, and say I’m skipping

• Multi-agent coordination. Multiple agents collaborating on one task (one writes code, another reviews it). Real and interesting. Out of scope — it’s a separate orchestration layer on top of this one.
• Fine-tuning or training agents. We’re orchestrating hosted models, not training custom ones.
• The prompt engineering. How to write good tool descriptions, how to structure the system prompt for reliability. Important for quality but not an architecture question.
• Agent marketplace / plugin ecosystem. How third parties publish tools. A product concern, not an infrastructure one.

11. Wrap-up

One crisp sentence:

This design treats agent orchestration as a durable state machine with safety gates at every boundary — the architectural center is not the LLM call but the permission model that controls what the LLM’s outputs are allowed to do, which is what separates a demo from a deployable system.

That framing — naming safety as the architecture, not a feature — is what lands.

What separates SDE II from SDE III on this question

• SDE II draws a loop (LLM → tool → LLM), mentions tool calling, names a few tools, gestures at “we’d need to handle errors.”
• SDE III designs the state machine with explicit states and transitions, separates capabilities from permissions as the safety model, names checkpointing for crash recovery with the exactly-once semantics for side-effecting tools, addresses prompt injection as a concrete threat model, and commits to budget enforcement as a system-level kill switch rather than a model-level instruction.
• Staff/Principal frames the agent design as a portfolio of autonomy trade-offs: which tool categories justify human approval overhead, which require sandbox isolation, and what the runbook looks like when an agent takes an unexpected action at 3am. Proactively identifies the scaling inflection point where a per-task worker model breaks down (at ~100K concurrent tasks, the cost of maintaining durable state per task and the coordination overhead of task queuing changes the architecture). Asks “what does on-call do when an agent charges a customer’s credit card for the wrong amount?” — which surfaces the need for compensating transactions, audit trails, and rollback tooling as architectural requirements. Surfaces the missing requirement of regulatory exposure: agents acting on behalf of users in financial or medical contexts may require explainability guarantees and data retention that materially constrain the checkpoint and logging design. Reasons about build vs buy for the workflow engine (Temporal vs home-built state machine) in terms of team operational burden, not just feature parity.

The difference isn’t knowing that agents call tools. It’s naming controlled autonomy as the design problem and showing the specific mechanisms (safety gates, checkpoints, sandboxing) that make it safe enough to deploy.

Further reading

• Designing Data-Intensive Applications — Kleppmann. Chapter 11 (stream processing) and Chapter 12 (future of data systems) are relevant to the event-driven state machine.
• System Design Interview Vol. 2 — Alex Xu. Useful for the general serving shape.
• Temporal.io documentation — the workflow engine model that inspires the checkpoint-and-resume pattern.
• Anthropic tool use documentation — the canonical reference for how LLM tool calling works at the protocol level.
• calm.rocks: Designing an LLM Serving Platform — the inference layer beneath the agent orchestrator.
• calm.rocks: Designing a RAG System — RAG is often one of the tools an agent calls; this walkthrough covers the retrieval tool in depth.