What it is
Two designs sit behind the word “message queue”. A work queue (SQS, classic RabbitMQ) tracks per-message state and deletes a message once a consumer acknowledges it. A partitioned log (Kafka) only appends; consumers remember their own position (an offset) per partition, and nothing is deleted on consumption, only by retention. The difference decides whether replay, fan-out to many consumer groups, and very high throughput are properties of the design or features bolted on.
Important
Key takeaway: A queue deletes what it delivers; a log remembers what it appended. Choose a work queue when messages are jobs that need per-message retry and dead-lettering at modest volume, and a log when throughput, replay, or several independent consumers dominate.
When you care
“Why Kafka and not SQS?” (or the reverse) is a standard follow-up to any asynchronous design, and “what if a consumer needs to reprocess last week?” is the question that separates the two models in one sentence.
Note
Interview signal:
Naming the third option, a database table used as a queue (SELECT ... FOR UPDATE SKIP LOCKED), as the right answer up to about
a thousand messages a second shows judgment. It is a credibility point, not a mistake, because it adds no new infrastructure.
The three options
| Dimension | Work queue | Partitioned log | Database table |
|---|---|---|---|
| Throughput ceiling | Per-message state: roughly 10K to 100K per second | Sequential append: millions per second across partitions | Row locks: about 1K per second |
| Replay | Impossible (delivered means deleted) | Rewind the offset; free | Possible, but the table bloats |
| Several independent consumer groups | One queue per group; the producer writes each message G times | Free: every group reads the same bytes with its own offsets | A status column per group |
| Ordering | None under competing consumers | Guaranteed within a partition | Transaction order, one consumer only |
| Per-message retry and dead-letter | Native (visibility timeout) | Built in client logic | Native (status columns) |
| Broker complexity | High: a state machine per message | Low: append, replicate, serve | None, it is your database |
How a log works
A topic is split into partitions, each an ordered, append-only sequence of messages with increasing offsets.
Publish.
- The producer computes the partition:
hash(key) mod partition_count. The same key goes to the same partition, which is the only source of per-key ordering. - The producer batches messages per partition (waiting a few milliseconds and compressing). Batching is why a million small messages a second is feasible: the broker sees thousands of large sequential writes.
- The partition leader appends the batch and assigns offsets; followers fetch and append.
- With
acks=all, the leader answers once every in-sync replica has the batch. The in-sync set is the durability contract. - A producer retry after a timeout carries a sequence number, so the leader discards duplicates (publish-side idempotence).
Consume.
- Consumers join a consumer group; a coordinator assigns partitions. One partition belongs to exactly one consumer in a group, so no two workers race on a key’s messages.
- Each consumer pulls from its offset (a long poll gives push-like latency without push’s problems).
- After processing, the consumer commits its offset. Commit after processing gives at-least-once delivery: a crash between the two re-fetches the batch.
- Replay is a client act: reset the committed offset (or look up by timestamp) and read forward. The bytes were always there.
- When a consumer joins or dies, the coordinator rebalances the partitions.
A group’s entire state is a map of partition -> offset, which is why fan-out is free and replay is trivial.
Worked example: parallelism ceiling. A topic with 4 partitions can keep at most 4 consumers in a group busy. A 10-consumer group leaves 6 idle. The partition count is the parallelism ceiling, fixed at topic creation, so plan it for peak consumer parallelism.
Caution
Common mistake:
“Just add partitions” when consumers fall behind. Changing the partition count changes hash(key) mod N, so keys remap and
per-key ordering breaks across the boundary. Size partitions for future parallelism up front, and scale consumers only up to that.
Backpressure and lag
A log makes backpressure structural: consumers pull, so a slow consumer fetches less often and nothing piles up in broker memory; unread messages simply stay in the log. What needs watching is lag, the gap between the head of the log and the group’s committed offset.
Warning
Production reality: Lag has a hard deadline. If lag exceeds retention, the oldest messages are deleted before the consumer reads them, so a slow consumer silently loses data while every broker is healthy. Alert on lag as a fraction of retention, not on absolute lag.
Hot partitions. hash(key) spreads keys, not load: one chatty key concentrates on one partition and caps that key at a single
broker’s append rate. Options: widen the key (user_id to user_id:session_id) where ordering is only needed at the finer grain;
route the known-hot key to a dedicated partition set; or provision for it, since sometimes a key’s ordering cannot be split.
Retention and compaction
| Model | Keeps | Serves |
|---|---|---|
| Time or size retention | Messages for a window (for example 7 days) | Event streams; replay bounded by the window |
| Log compaction | The latest message per key, forever | A durable changelog: replay from offset zero and reconstruct current state per key |
Compacted topics are how consumer offsets themselves are stored, and the bridge to “the log as a database”.
When to pick what
| Situation | Choose | Why |
|---|---|---|
| Jobs with per-message retry, visibility timeout, dead-letter queue (resize this image) | A work queue | Per-message state is exactly what you want |
| Event stream, many consumers, replay, very high throughput | A partitioned log | Sequential I/O, offsets per group |
| Under about 1K messages a second, already have a database | A table with SKIP LOCKED | No new infrastructure |
| Per-key ordering | A log, with the key as the partition key | Order within a partition |
| Rebuild a downstream store from history | A log with long retention or compaction | Replay |
Related
- Walkthrough: Designing a Distributed Queue — the full design of the broker this reference compares.
- Walkthrough: Designing an Ad Click Aggregation System — a partitioned log as the system of record for a high-volume aggregation pipeline.
- Reference: Queue Delivery Semantics — at-most-once, at-least-once, and effectively-once on top of either model.