What it is
Delivery semantics define how many times a message published to a queue is processed by a consumer when failures happen: at most once, at least once, or exactly once. The guarantee is not a queue feature you switch on — it emerges from where acknowledgments happen relative to processing.
When you care
Any design with a queue gets this follow-up: “What happens if the consumer crashes halfway through?” The trap is answering “we’ll use exactly-once delivery” as if it were a checkbox — interviewers know it’s the hardest guarantee in distributed systems and probe whether you know what it actually requires. The strong answer names the failure window, picks at-least-once, and makes processing idempotent.
The semantics
| Guarantee | Mechanism | On consumer crash | Use when |
|---|---|---|---|
| At-most-once | Ack/commit before processing | In-flight message lost | Data loss beats duplication: metrics, telemetry, presence pings |
| At-least-once | Ack/commit after processing | Message redelivered → duplicates | The default — paired with idempotent processing |
| Exactly-once | At-least-once + dedup/transactions end-to-end | Redelivered but deduplicated | Within one framework’s boundary; expensive across systems |
At-most-once is just commit-first ordering. Crash after commit, before processing: the message is gone. Nobody picks this deliberately for business data; everybody gets it accidentally by auto-committing offsets on fetch.
At-least-once is commit-last ordering. Crash after processing, before commit: the message is processed again. The failure window between “done” and “acknowledged” can’t be closed — only handled. This is the right default, because duplicates are handleable and loss is not.
Exactly-once is at-least-once with the duplicates made invisible. The phrase hides a critical distinction: exactly-once delivery (the consumer sees the message once — impossible to guarantee under crashes) vs exactly-once processing (the effect happens once — achievable). Kafka’s transactions achieve it within a consume-process-produce pipeline by committing offsets and outputs atomically; the guarantee evaporates the moment a side effect leaves that boundary (an email, an external API call).
Idempotency: the other half of at-least-once
At-least-once is only safe if reprocessing is harmless. Standard patterns:
| Pattern | Mechanism | Cost |
|---|---|---|
| Natural idempotence | Operation is a set/upsert (status = shipped) | Free — prefer it where possible |
| Idempotency keys | Consumer records processed message IDs; skips repeats | A dedup store with TTL ≥ redelivery window |
| Conditional writes | UPDATE ... WHERE version = n / compare-and-set | DB does the dedup; needs versioned rows |
| Transactional outbox | Effect and “processed” marker commit in one DB transaction | The strongest pattern; one DB round-trip |
The producer side has its own duplicate problem — retrying a timed-out publish can append twice. Producer idempotence (per-producer sequence numbers, as in Kafka) or publish-side idempotency keys close it.
Replay changes the question
In log-based systems (Kafka-style), messages aren’t deleted on consumption — consumers can rewind offsets and reprocess history deliberately. Replay turns “duplicates are a failure case” into “reprocessing is a feature”: backfills, bug fixes, and new consumers all reprocess old messages on purpose. A consumer designed for at-least-once (idempotent effects) gets replay safety for free — one more reason idempotence, not exactly-once machinery, is the foundation to build on.
When to pick what
- Default: at-least-once + idempotent consumers (transactional outbox or idempotency keys).
- Loss-tolerant, high-volume telemetry: at-most-once — and say the loss is acceptable out loud.
- Stream pipeline fully inside one framework: that framework’s exactly-once transactions (Kafka, Flink).
- Side effects on external systems: there is no exactly-once — idempotency keys on the external call are the only defense.
Related
- Walkthrough: Designing a Distributed Queue — where commit timing lives in the consume path, and producer idempotence in the publish path.
- Walkthrough: Designing an Ad Click Aggregation System — exactly-once semantics in a streaming aggregation pipeline.
- Walkthrough: Designing an Order Processing System — idempotency keys and sagas where the side effects are payments.