What it is
An operation is idempotent if doing it twice has the same observable effect as doing it once. In a distributed system, retries are not an edge case: networks drop, mobile clients resend, a message queue redelivers after a crash, and a user double-clicks. Since exactly-once delivery is not achievable, correctness comes from making repeated processing harmless. The outbox pattern solves the neighboring problem of updating a database and publishing a message without losing either.
Important
Key takeaway: Idempotency must hold at every layer a retry can cross: client to gateway, gateway to service, service to each downstream call. A retry that crosses three services with three different mechanisms will eventually produce a duplicate.
When you care
Any money, order, or message flow gets “what if the request is retried?” The cheapest strong answer is an idempotency key, and the strongest follow-up is explaining how it propagates.
Note
Interview signal: Order processing has the worst idempotency surface in e-commerce: a duplicate retry is a double charge, and the support cost of one double charge exceeds the cost of the whole idempotency layer. Saying so, and that the infrastructure is cheap, is the senior framing.
Patterns
| Pattern | Mechanism | Cost |
|---|---|---|
| Naturally idempotent operation | A set or upsert (status = shipped), not an increment | Free; prefer it |
| Idempotency key | The client sends a unique key per logical operation; the server records the result under the key and returns it on repeats | A dedup store with a time-to-live at least as long as the retry window |
| Conditional write | UPDATE ... WHERE version = n (compare-and-set) | The database dedups; rows need a version |
| Unique constraint | A key that the database refuses to insert twice | The cheapest dedup, if the key is natural |
| Transactional outbox | The effect and the “processed” marker commit in one local transaction | One database transaction |
Idempotency keys end to end
A key is minted where the user’s intent is formed and carried through every hop.
| Hop | Key | Notes |
|---|---|---|
| Client to gateway | A client-generated UUID, one per place-order attempt, sent as an Idempotency-Key header | Generate once per attempt, reuse on every retry of that attempt |
| Gateway to order service | The same key | Do not mint a new one |
| Order service to workflow | The order id, derived deterministically from the key | One request maps to one order |
| Workflow to each downstream call | A key derived from (order_id, step_name) | Retrying a step three times sends the same key, so the payment provider returns the same authorization |
| Each downstream service | Stores keys it has seen for about 24 hours and returns the cached response on a duplicate | The window must exceed the longest retry horizon |
Worked example: handling a duplicate. The server receives POST /orders with key abc-123. It looks up abc-123: if it has a
completed result, return it; if it is in progress, return a conflict or wait (two concurrent retries must not both execute);
if it is unseen, record abc-123 -> in progress atomically (a unique insert), do the work, store the result, and return it.
The atomic first step is the point: without it, two simultaneous retries both pass the lookup and both execute.
Mismatched reuse (the same key with a different payload) should be rejected, not silently served the old result.
Caution
Common mistake: Checking for the key with a read, then writing it with a separate write. The two-step check is a race. Make “claim the key” a single atomic operation (a unique constraint or a conditional put) and treat the loser as a duplicate.
The dual-write problem and the outbox
A service often must update its database and publish an event (“order created”). Doing both separately is unsafe: write the database then publish, and a crash in between loses the event; publish then write, and a crash leaves an event for something that never happened. There is no transaction that spans a database and a message broker.
The transactional outbox removes the dual write:
- In one local database transaction, write the business row and insert an event row into an outbox table.
- A separate relay reads new outbox rows and publishes them to the broker. It deletes or marks a row only after the broker acknowledges.
- Consumers receive events at least once (the relay can crash after publishing and before marking), so consumers are idempotent.
Two ways to implement the relay: polling the outbox table on an interval (simple, adds latency and query load) or change data capture that tails the database’s transaction log (low latency, more infrastructure). Either way the event is guaranteed to be published if and only if the transaction committed.
Consumer-side counterpart: the inbox. The consumer records processed message ids in the same transaction as its effect, so processing a redelivered message is a no-op. Outbox on the sender and inbox on the receiver together give effectively-once processing over at-least-once delivery.
Warning
Production reality: Idempotency stores need a time-to-live, and the time-to-live is a promise: a retry that arrives after the key has expired is processed again. Set it longer than the longest realistic retry horizon (including a queue backlog and a dead-letter replay), and size the store for that many keys.
When to pick what
| Situation | Choose |
|---|---|
| A simple status update | Make it naturally idempotent (set, not increment) |
| An API that creates things or moves money | An idempotency key, claimed atomically |
| Rows already have a natural unique key | A unique constraint |
| Update a database and publish an event | Transactional outbox |
| Consume at-least-once messages with side effects | An inbox of processed ids, or natural idempotency |
| Calling an external API with no idempotency support | There is no exactly-once; use their key if offered, and reconcile |
Related
- Walkthrough: Designing an Order Processing System — an idempotency key that flows from the client through every saga step.
- Reference: Distributed Transaction Patterns — sagas and compensation, which depend on every step being safe to retry.
- Reference: Queue Delivery Semantics — why at-least-once delivery is the default, and idempotency is what closes the gap.