What it is
Stream processing computes results over an unbounded flow of events as they arrive, instead of over a finished file. Four ideas come up in every streaming design. Windows cut the endless stream into finite groups to aggregate. Event time says which clock a window is cut by. Watermarks say when a window is complete enough to emit. And exactly-once is a property of the whole pipeline, not a switch on one component.
Important
Key takeaway: Window by event time, not arrival time; use a watermark to decide when to emit and an allowed lateness to bound how long you wait; send what arrives later to a side output, not the void. “Exactly-once” needs three components advancing together: deduplication, checkpointed state, and a transactional sink.
When you care
Ad-click aggregation, metrics, fraud detection, and real-time dashboards all reduce to “count things per window”. The follow-ups are always the same: what about late events, what about duplicates, what about a hot key, and how do you fix wrong numbers?
Note
Interview signal: Most candidates name exactly-once as a requirement. Naming the three pieces (event-id dedup, checkpoint barriers, transactional sink) shows you know it is a property of the system, not a setting. Adding a batch reconciliation path shows you know the stream can still be wrong.
Windows
| Window | Definition | Example |
|---|---|---|
| Tumbling | Fixed, non-overlapping intervals | Clicks per ad per minute |
| Sliding | Fixed length, advancing by a smaller step; windows overlap | Average over the last 5 minutes, updated every minute |
| Session | Closed by a gap of inactivity per key | A user’s browsing session |
A sliding window of length L and step s puts each event in L ÷ s windows, so a 5-minute window sliding every 1 minute does five times the work of a tumbling one. Choose the coarsest window that answers the question.
Event time, processing time, and watermarks
- Event time is when the event happened (a timestamp in the event). Processing time is when the system sees it. They differ by network delay, retries, offline devices, and backlogs.
- Windows that use processing time are simple and wrong whenever there is lag. Windows by event time are correct but must decide when a window is finished, because late events may still arrive.
- A watermark is the system’s estimate of “no more events older than time T will arrive”. It is the clock that tells the engine a window is done enough to emit. It advances as new events are seen, minus a configured out-of-orderness slack.
- Allowed lateness keeps a closed window’s state around for a while longer so late events can update the result.
Worked example. A one-minute tumbling window 10:00 to 10:01 with one hour of allowed lateness:
| Watermark | What happens |
|---|---|
| Reaches 10:01 | The window first emits a result (the “on-time” answer) |
| Between 10:01 and 11:01 | A late event with event time 10:00:30 arrives; the window updates and re-emits a corrected result |
| Passes 11:01 | The window’s state is dropped; any later event for it goes to the side output |
Events beyond the allowed lateness go to a separate topic (a side output), not silently dropped, and a batch job reads that topic too so nothing is permanently lost, only delayed. The trade-off is memory and freshness: more lateness means more state held and more corrections downstream.
Exactly-once is a property of the system
Duplicates come from retried API calls, client SDK retries, and a processor restarting from a checkpoint. Three pieces together prevent double-counting:
- Deduplicate by event id in the processor’s state, before events enter any window. This handles logical duplicates (the same click sent twice).
- Checkpoint barriers. The processor periodically snapshots, atomically, its input position (the log offset), all in-flight window state, and the output position. After a crash it restarts from the last snapshot and replays only the events after it.
- A transactional sink. The write to the output store is part of the same commit as the checkpoint. Without it, a crash between “checkpoint succeeded” and “write succeeded” would skip or double events.
Miss any one and the guarantee does not hold. Exactly-once holds inside the pipeline’s boundary; side effects that leave it (an email, a call to an external API) cannot be un-sent, and need idempotency keys instead.
Hot keys in a stream
The input is partitioned by key (an ad id) and the processor’s keyed state uses the same key, which scales linearly with
partitions. But hash(key) spreads keys, not load: during a big event one ad can carry 10% of all clicks, pinning one partition at
100% while the rest idle.
Salting with a two-stage aggregation. Detect the hot keys (the top 0.01% by volume, with an approximate counter), and for those
use key + random suffix (0..N) as the partition key. A first stage aggregates each salted sub-key; a second stage, partitioned on the
unsalted key, sums the N partial results per window. It costs one extra shuffle and keeps the hot partition from melting. N is
chosen per key from observed load; 4 to 16 is usually plenty.
Lambda and Kappa: owning freshness and correctness separately
A streaming path is fast but has failure modes it cannot see itself: events later than the allowed lateness, a bug in the dedup logic, a job restarted from a stale checkpoint. The Lambda answer is a second path:
- A batch job over the raw events (kept in cheap storage) recomputes completed windows, for example hourly, and overwrites the streaming results for them.
- Readers of old buckets (billing) always read reconciled data; readers of the fresh tail (dashboards) read the streaming results, “probably right within seconds”.
The streaming path can then be wrong and the system is still eventually correct. Kappa replaces the batch path with a replay of the same stream through the same code: one pipeline and one data model, simpler to operate, but it asks the stream to be the system of record. Where correctness is strict (money) and batch compute is cheap, an explicit batch layer is often worth its weight.
Warning
Production reality: A silently wrong aggregate is worse than a missing one. Whichever architecture you choose, build the reconciliation and say how a number gets corrected after the fact, and watch for the symptoms of a stuck pipeline: a watermark that stops advancing, a consumer group whose lag grows, and window state that grows without bound.
When to pick what
| Need | Choose | Why |
|---|---|---|
| Counts per fixed interval | Tumbling event-time windows | Simplest correct aggregation |
| A rolling metric | Sliding windows, coarsest step that works | Cost scales with overlap |
| Per-user activity bursts | Session windows | Gap-based boundaries |
| Out-of-order or mobile events | Event time, a watermark, allowed lateness | Arrival order lies |
| Events later than the lateness bound | Side output plus a batch correction | Nothing is dropped silently |
| Duplicates from retries | Event-id dedup plus checkpointed state | Logical and replay duplicates |
| A skewed key | Salting with two-stage aggregation | Spreads a hot partition |
| Strict correctness (billing) | A batch reconciliation layer over raw events | The stream cannot see its own errors |
Caution
Common mistake: Windowing by processing time and calling the numbers real-time. During a backlog, a catch-up burst lands in one window and the trend is wrong. Window by event time, and say what your watermark and allowed lateness are.
Related
- Walkthrough: Designing an Ad Click Aggregation System — windows, exactly-once, and batch reconciliation in a pipeline that bills money.
- Walkthrough: Designing a Top K System — sliding-window aggregation feeding a heavy-hitters structure.
- Reference: Approximate Counting and Membership — the sketches you put inside a window when exact state is too big.