Walkthrough: Designing an Order Processing System

The problem

You’re asked to design an order processing system — Amazon-style: “A user has items in their cart. They click ‘place order’. We need to charge their card, reserve inventory across multiple warehouses, route to shipping, send a confirmation email, and update the user’s order history.”

It looks like a CRUD app. It isn’t. The hidden trap is that every step can fail independently, and a half-completed order is worse than no order — it produces phantom charges, unreservable inventory, or shipped goods to the wrong address. The question is testing whether you understand distributed-transaction tradeoffs in the real world: 2PC won’t work at scale, so how do you build correctness from cooperating eventually-consistent services?

Below is how I’d walk through this.

1. Clarify before you design

First 4–6 minutes.

• Scale. Orders per second at peak? Average cart size? (For Amazon-scale, ~5K orders/sec average, ~50K/sec on Prime Day.)
• Inventory model. Single warehouse, multi-warehouse, or third-party-fulfillment? (Multi-warehouse changes everything — reservation must pick warehouses.)
• Payment. Credit card only, or multiple methods (Apple Pay, PayPal, gift cards)? Stored cards or one-shot?
• Order state. What states does an order pass through? (I’d expect: pending → paid → reserved → fulfilling → shipped → delivered, with cancelled/refunded as side branches.)
• Cancellation. Can a user cancel? Up to which state?
• Consistency window. When the user clicks “place order,” do they need an immediate confirmation, or is “we’ll email you in a few seconds” acceptable? (Affects whether place-order is sync or async.)
• Idempotency. Mobile retries are a guaranteed source of duplicate orders. Is the client expected to send an idempotency key?

Say the interviewer confirms: 5K orders/sec average / 50K peak, multi-warehouse fulfillment, cards + Apple Pay + gift cards, the state machine above, cancellation up to “fulfilling,” 1–2 second confirmation latency acceptable, mandatory idempotency key from client.

2. Capacity estimate

• Order writes: 5K/sec → 432M orders/day average; 4.3B/day Prime Day peak. At ~2KB per order (header + items + addresses), that’s ~860 GB/day average ingest, ~8 TB/day peak.
• Read traffic: Order history page, order status page. ~10× the write rate → 50K reads/sec average. (Most of this is cacheable per-user.)
• Inventory ops: Each order touches an average of 2.5 SKUs. ~12.5K reservation ops/sec average; 125K/sec peak.
• Payment ops: External calls to payment gateway. 5K/sec average. Latency from the gateway dominates the order path (200–800ms typical).
• Storage growth: Orders are kept indefinitely (legal / customer-history). At the order-state level, ~1 PB after 5 years.

The takeaway: “This isn’t a single-database design. The order record’s life involves at least four distinct services (payment, inventory, shipping, notifications) — each with its own latency and failure profile. The architectural question is how to coordinate them without a global transaction.”

3. API design

POST /api/orders
  body:    { cart_id, user_id, payment_method_id,
             shipping_address_id, idempotency_key }
  returns: { order_id, status: "accepted" | "duplicate" }

GET /api/orders/{order_id}
  returns: { order_id, status, items, totals, timeline[] }

POST /api/orders/{order_id}/cancel
  body:    { reason, idempotency_key }
  returns: { status: "cancelled" | "too_late" }

GET /api/users/{user_id}/orders?cursor
  returns: { orders[], next_cursor }

Two non-obvious decisions:

1. POST /orders returns 202 Accepted, not 200 OK. The response carries the order_id and a “we’re processing” status. Final confirmation arrives via the order detail page (polled or pushed via WebSocket) within 1–2 seconds. This decouples the API from the slow downstream calls.
2. Idempotency key required at the gateway. Same key + same user within 24 hours returns the same order_id deterministically. This is the only defense against the “user clicked place order twice” bug, which is otherwise endemic.

4. Core design choice: orchestration vs choreography

This is the question. The decision is how the order flows through its lifecycle across services.

(a) Synchronous monolith

A single service does everything in one request: charge the card, decrement inventory, queue shipping, send email. One database transaction.

• Pros. Simple. ACID gives correctness.
• Cons. Catastrophic at scale — 200ms payment + 50ms inventory + 100ms shipping + 30ms email = ~400ms per order before any retries, and any one failure poisons the whole request. Throughput capped by the slowest dependency.

(b) Distributed transaction (2PC)

Two-phase commit across payment, inventory, and shipping services.

• Pros. ACID across services if all participants implement it.
• Cons. External services (payment gateways) don’t support 2PC. Coordinator failures leave participants blocked. The pattern is widely understood as the wrong tool for e-commerce flows.

(c) Saga pattern

Each step is a local transaction in its own service. A coordinator (orchestration saga) or events (choreography saga) sequence them. Each step has a compensating action that undoes it if a later step fails.

• Pros. No global lock. Each service owns its own consistency. Compensations are explicit and inspectable.
• Cons. Eventual consistency by construction. The compensation logic is real work — you must define how to refund a charge, release inventory, cancel a shipment.

Pick: orchestration saga

The right answer for this problem. The reasoning, specific to Amazon-style order flow: the steps are not symmetric — payment has different failure semantics than inventory than shipping — so a central orchestrator makes the saga’s state explicit and debuggable. Choreography (services emitting and reacting to events) loses the global view; on Black Friday you absolutely want a central log saying “order 12345 is in step 3, waiting for warehouse-east inventory ack.”

The saga, step by step:

1. Reserve inventory      (compensation: release inventory)
2. Authorize payment      (compensation: void authorization)
3. Persist order          (compensation: mark cancelled)
4. Confirm payment capture (compensation: refund)
5. Notify shipping        (compensation: cancel shipment)
6. Send confirmation email (no compensation needed — best effort)

Inventory before payment is deliberate: a payment hold is cheap and reversible; inventory is the scarce resource. We fail fast on out-of-stock without an unnecessary auth on the card. (Some platforms reverse this; the choice depends on which resource is more contested.)

I’d say out loud: “Saga, orchestrated, with inventory before payment because inventory is the scarce resource and payment auth is cheap to reverse. The orchestrator owns the saga state and is the single source of truth for ‘where is my order’.”

5. Data model and storage

orders (RDBMS, sharded by user_id):
  order_id (PK), user_id, status, total, currency,
  created_at, updated_at, shipping_address_id,
  payment_method_id, idempotency_key, version

order_items (RDBMS, co-located with orders):
  order_id, sku_id, qty, unit_price, warehouse_id

saga_state (RDBMS):
  saga_id (PK), order_id, current_step, step_history[],
  status, started_at, last_updated_at

idempotency_keys (KV, e.g., DynamoDB):
  key (user_id + idempotency_key) → order_id
  TTL: 24 hours

order_events (append-only log, e.g., Kafka → Cassandra):
  event_id, order_id, event_type, payload, occurred_at

Why these choices:

• orders and order_items in a relational store. The order is the unit of consistency (header + items must be written together). Postgres or MySQL handles this; sharded by user_id so a user’s order history is a single-shard read.
• saga_state separately. The orchestrator’s state is high-write (every step transition) and is logically distinct from the order. Putting it on the same DB row as the order causes contention — the order is the read-heavy artifact for customer support; the saga is write-heavy for the orchestrator.
• idempotency_keys in a KV store. A simple lookup with TTL. Doesn’t deserve a relational store.
• order_events as an append-only log. Every state transition emits an event. This becomes the source of truth for analytics, customer-support timelines (“at 14:32 we reserved warehouse-east”), and re-deriving the order projection if needed. See the event-sourcing pattern.

Topic-specific reasoning: order data is write-once-then-mostly- immutable (state transitions are bounded; the 80% of fields never change after creation). That’s an excellent fit for a relational store with the events table providing append-only history — the relational store carries the projection, the events carry the audit log.

6. The hot path: the saga walkthrough

The orchestration flow:

flowchart LR
  Client[Client] --> Gateway[API gateway]
  Gateway -->|idempotent check| IdempKV[(Idempotency KV)]
  Gateway --> OrderSvc[Order service]
  OrderSvc --> Saga[Saga orchestrator]
  Saga --> InvSvc[Inventory service]
  Saga --> PaySvc[Payment service]
  Saga --> ShipSvc[Shipping service]
  Saga --> NotifSvc[Notification service]
  OrderSvc --> OrderDB[(Orders DB)]
  Saga --> SagaDB[(Saga state DB)]

Walking through one happy-path order:

1. Gateway receives POST /orders with idempotency_key. Looks up the key. Not found → assigns a new order_id and stores (key → order_id) with TTL. Found → returns the existing order_id immediately.
2. Order service writes a draft orders row in pending state, then asks the saga orchestrator to start.
3. Saga step 1: reserve inventory. Sends ReserveInventory(order_id, items[]) to the inventory service. Inventory service performs an atomic decrement on each SKU’s hot inventory shard (see the flash deal walkthrough for the contention pattern). Returns (reservation_id, warehouses[]) on success or OutOfStock(sku) on failure. Saga writes the step’s outcome to saga_state.
4. Saga step 2: authorize payment. Sends Authorize(order_id, amount) to the payment service, which calls the external gateway with an idempotency key derived from order_id. Returns (auth_id, token) or a failure code.
5. Saga step 3: persist final order. Updates orders.status to paid, attaches reservation_id and auth_id. From here, the user-visible order detail page returns “your order is confirmed.”
6. Saga step 4: capture payment. Calls the gateway to convert the auth to a capture. (Some flows defer capture to ship-time; the choice depends on legal/business policy.)
7. Saga step 5: notify shipping. Emits a FulfillOrder event to the shipping service’s queue. Shipping owns the lifecycle from there.
8. Saga step 6: send email. Async, fire-and-forget. Email delivery isn’t part of order correctness.

When the saga completes, saga_state.status = completed. The saga record is retained for audit but no longer actively driven.

The TTL/invalidation behavior worth naming:

• Idempotency keys: 24h TTL. Long enough for client retries; not so long that the KV grows unbounded.
• Inventory reservations: 30-minute TTL before saga step 2 completes. If payment auth fails, the reservation expires naturally. The compensating release is a defensive backstop.
• Saga state: retained 90 days, then archived. We need it for customer-support investigations.

7. The compensation path: what happens when step N fails

This is where SDE III answers diverge most. Each step has an explicit compensation; the saga walks them in reverse.

Two cases worth walking through:

Case A: payment auth fails (step 2). Saga has already reserved inventory in step 1. Compensation: call inventory’s Release(reservation_id). Order is marked failed with a user-facing reason (“your card was declined”). User sees a retry-with-different-card UI. Inventory is back in the pool.

Case B: shipping notification fails (step 5). Saga has already captured payment. Compensation is not an automatic refund — shipping failures are usually transient (the shipping service is restarting, the warehouse system is offline). The saga retries step 5 with exponential backoff up to 24 hours before giving up. Only after persistent failure does the saga walk back to compensate steps 4–1.

The principle: compensations are last-resort, retries are first-resort. Most “failures” in distributed systems are transient; treating every failure as terminal causes more bad outcomes than the failures themselves.

I’d say out loud: “I’d retry transient steps with bounded backoff before compensating. Compensation is for permanent failures only — out-of-stock, card permanently declined, address not deliverable.”

8. Idempotency: the most underrated part of this design

Mobile clients retry. Networks drop. Users double-click. Without idempotency, every retry under load creates a phantom order or a double charge. Idempotency must work at every layer, not just the API gateway.

The propagation of idempotency keys:

• Client → Gateway. Client-generated UUID per place-order attempt.
• Gateway → Order service. Same key.
• Order service → Saga. Saga’s order_id is its own idempotency key (deterministic from the original key).
• Saga → Each downstream call. Each call carries an idempotency key derived from (order_id, step_name). Retrying step 2 three times produces three identical auth requests with the same key — the gateway returns the same auth on each.
• Each downstream service. Stores recently-seen idempotency keys for 24h. Returns cached response on duplicate.

The full propagation, not optional anywhere: a retry that crosses three services with three different idempotency mechanisms is a retry that will eventually produce a duplicate.

Topic-specific reasoning: order processing has the worst idempotency surface in e-commerce — the $1,200 charge on a duplicate retry is the kind of failure that costs more in customer support than the order itself. The cost of idempotency infrastructure is low; the cost of not having it is huge.

9. Sharding and read paths

Orders DB is sharded by user_id. Per-user order history is a single-shard read; cross-shard analytics queries go to a warehouse, not the OLTP DB.

Inventory is sharded by sku_id with hot SKUs getting multi-shard treatment per the flash-deal pattern.

Saga state is sharded by saga_id. Each saga is owned by one shard; the orchestrator routes step events accordingly.

Read traffic for the order history page is heavy and cacheable per user. We add a Redis cache keyed on user_id with a 60-second TTL. Order updates invalidate (write-through) for the user whose order changed.

Extending the architecture diagram with sharding, caches, and the events stream:

flowchart LR
  Client[Client] --> Gateway[API gateway]
  Gateway -->|idempotent check| IdempKV[(Idempotency KV)]
  Gateway --> OrderSvc[Order service]
  OrderSvc --> Saga[Saga orchestrator]
  Saga --> InvSvc[Inventory service]
  Saga --> PaySvc[Payment service]
  Saga --> ShipSvc[Shipping service]
  Saga --> NotifSvc[Notification service]
  OrderSvc --> Cache[(Order cache)]
  Cache -->|miss| OrderDB[(Orders DB · sharded by user_id)]
  Saga --> SagaDB[(Saga state · sharded by saga_id)]

  OrderSvc --> Events[(Order events · Kafka)]
  Saga --> Events
  Events --> Warehouse[(Analytics warehouse)]
  Events --> Search[(Order search index)]

  classDef new fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class Cache,Events,Warehouse,Search new

The events stream feeds three downstream consumers:

• Analytics warehouse. Reporting, fraud models, demand forecasting.
• Order search index (Elasticsearch). Customer-support queries: “find me all orders from this user with status pending in the last week.”
• Reconciliation. Each midnight, a job replays the day’s events and verifies all four services agree on each order’s state. Drift is alerted, not auto-corrected.

10. Failure modes

Name what fails, name what degrades gracefully, name what doesn’t.

• Payment gateway is slow / down. Saga step 2 retries with backoff up to 30 seconds. After that, order is marked failed, inventory released, user shown “try again.” We do not hold inventory open for hours waiting on a stuck gateway — that’s how Black Friday goes wrong.
• Inventory service is down. Saga step 1 fails. Order is rejected at the API with 503 Service Unavailable. The user’s cart is preserved; the place-order button is disabled with a clear message. (Better than queuing internally — telling the user honestly is correct.)
• Saga orchestrator crashes mid-saga. Saga state is durably written to its DB after every step; on restart, the orchestrator resumes from the last persisted step. The saga’s event-driven resumption is the load-bearing correctness mechanism. Sagas cannot be in-memory state machines.
• Duplicate place-order from client. Idempotency key catches this at the gateway; same order_id returned. No phantom order.
• Race: user cancels at the same moment shipping is dispatching. This is real and visible at scale. Resolution: cancellation goes through the saga and attempts to compensate step 5. If shipping returns “already shipped, can’t cancel,” the cancellation fails and the user sees “your order has shipped; please return on receipt.” We don’t ship-and-refund silently.
• Eventual consistency between order DB and search index. Events stream is the consistency mechanism. Lag is monitored; alerted at >30 seconds. Customer-support tools must annotate “this view may be 5 seconds stale.” Honesty beats a pretense of strong consistency we don’t actually have.

11. What I’d skip, and say I’m skipping

Time check: a few minutes left. Things I’d defer with one sentence each:

• Tax calculation. A separate service (Avalara, internal tax svc), called as part of cart pricing — not part of the order saga.
• Recommendation rails on the post-purchase page. Off the hot path.
• Multi-currency / FX. Real complexity, but downstream of pricing. Ask for it explicitly if scoping.
• Returns and reverse logistics. A separate saga that starts when a return is requested. Skipping unless asked.
• Gift cards as payment. A second payment service call before card auth; the saga adds steps but the pattern is the same. Defer unless the interviewer probes.
• Subscription / recurring orders. That’s a different product entirely, not an extension of one-shot orders.

12. Wrap-up

This design treats an order’s lifecycle as an explicit saga with durable orchestrator state, eventual consistency between services, mandatory idempotency keys at every hop, and compensations only as the last resort after retries fail. The single biggest commitment is “no global transactions” — every service owns its own consistency, and the saga state is the only thing that knows the global view.

That’s the answer that lands the round.

What separates SDE II from SDE III on this question

• SDE II picks a saga, names ACID, draws the four services, and mentions retries.
• Staff/Principal opens by identifying the missing requirements that would change the architecture: “what’s the policy when inventory is reserved but payment is never attempted — 30-minute TTL, or does customer support manually resolve? That changes whether the compensation is automatic or human-in-the-loop.” Proactively maps scaling inflection points: “this saga design holds to ~50K orders/sec; beyond that, the saga orchestrator’s DB becomes the bottleneck and you’d need to shard saga state by order range — that’s a non-trivial migration.” Thinks operationally: “on Black Friday at 3am, when saga ID 8472991 is stuck in step 3 with no forward progress and the customer is calling support, what does on-call look at, and does the tooling exist to manually advance or compensate a saga?” Surfaces the build-vs-buy question on the orchestration layer: a Temporal or Conductor deployment is a 2-week integration; a homegrown saga table is a 2-sprint build with years of edge-case handling ahead. Asks “when does idempotency break?” — if the derived key for a downstream call collides across two genuinely different orders, you get a phantom idempotency hit — and names the mitigation: use a cryptographic hash of the full order context, not just (order_id, step_name).
• SDE III commits to orchestration over choreography with a specific reason, walks the compensation path explicitly, propagates idempotency keys through every hop, treats inventory- before-payment as a deliberate ordering decision, and names the reconciliation pipeline as a peer concern.

The difference: seeing the order as a long-lived saga with real failure modes, not as an HTTP request that calls four services.

Further reading

• Designing Data-Intensive Applications — Kleppmann. Chapters 7–9 (transactions, consistency, distributed systems failure modes).
• System Design Interview Vol. 2 — Alex Xu. Chapter on the order/payment system is the canonical textbook walkthrough.
• Saga pattern — the original pattern reference; the orchestration vs. choreography distinction is from this source.
• Martin Fowler: Event Sourcing — the rationale for the order_events log.
• Walkthrough: Designing a Flash Deal System — the inventory contention pattern referenced here.
• Walkthrough: Designing a Notification System — the email-confirmation step’s downstream system.