Walkthrough: Designing a Notification System

The problem

You’re asked to design a notification system — the infrastructure behind push notifications, emails, SMS, and in-app messages for a large-scale product. The interviewer says something like: “Design a system that sends hundreds of millions of notifications per day across multiple channels, with user preferences and deduplication.”

Sounds like “a queue with a sender.” It isn’t. This is the canonical “fan-out meets prioritization meets user state” problem. The trap is that a notification system has to balance three forces simultaneously: throughput (millions of events per hour), user preferences (each user has channel preferences, quiet hours, and opt-out rules), and deliverability (each channel has its own rate limits, retry semantics, and failure modes). The interviewer is watching for whether you treat this as a message queue or as a stateful routing engine.

Important

The core tension: Priority isolation is the binding constraint, not throughput. A single queue that blends OTPs and marketing emails guarantees that a marketing blast can delay security alerts. The entire architecture — separate queues per channel and priority, dedicated workers per channel — exists to enforce structural isolation so critical notifications never wait.

Below is how I’d walk through this, roughly in the order I’d speak the words.

1. Clarify before you design

First 3–5 minutes. Resist the urge to start drawing.

Questions I’d ask:

Channels. Which channels? Push (iOS/Android), email, SMS, in-app, webhook? Each has radically different delivery semantics.
Scale. How many notifications per day, how many users? A million notifications is a queue; a billion is an architecture.
Latency requirements. Are some notifications time-critical (security alerts, OTPs) and others batch-tolerant (marketing digests)? The priority model changes everything.
User preferences. Can users configure per-channel opt-in/opt-out, quiet hours, frequency caps? Where does this state live?
Deduplication. If an event fires twice (retry, producer bug), should the user see it twice?
Template vs raw. Do senders provide the full message or do they provide event data and the system templates it per channel?
Who triggers notifications? Internal services via API? Or also user-configurable rules (e.g., “notify me when stock drops below X”)?

Say the interviewer confirms: four channels (push, email, SMS, in-app), 500M notifications/day, some are real-time (< 30s delivery for OTPs and security alerts), most are near-real-time (< 5 min), user preferences stored and respected, deduplication required, templated per channel, triggered by internal services via an API.

2. Capacity estimate

Brief. The point is to check the scale shape.

500M/day ≈ 6,000 notifications/sec average, ~50K/sec peak (morning surge, marketing campaigns).
Storage: user preferences for 500M users at ~500 bytes each ≈ 250GB. Notification history for audit/debug: 500M/day × 365 × ~200 bytes ≈ 36TB/year.
Channel-specific: push and in-app are our own infrastructure. Email and SMS go through third-party providers (SES, Twilio) with their own rate limits.

I’d say out loud: “This tells me two things. One — the throughput is high enough that per-notification synchronous lookups (preferences, templates, dedup) need to be fast — sub-millisecond — so those data stores need to be in-memory or heavily cached. Two — the system’s throughput ceiling is set by the external providers (email, SMS rate limits), not by our own infrastructure. Backpressure management is a first-class concern.”

3. API design

One endpoint ingests events; the system decides what to send where.

POST /v1/notifications
  body: {
    event_type: string,
    user_ids: [string] | segment_id,
    data: { ... },
    priority: "critical" | "high" | "normal" | "low",
    idempotency_key: string
  }
  returns: { notification_id, status: "accepted" }

GET /v1/notifications/:id/status
  returns: { channels: [{ channel, status, delivered_at }] }

Decisions worth saying out loud:

Accepts user_ids or a segment_id. A marketing campaign targeting 2M users shouldn’t require the sender to enumerate 2M IDs. The system resolves the segment internally — fan-out is the system’s job, not the caller’s.
Priority is explicit in the request. Not inferred from event_type. The same event might be critical for one product surface and low for another. Making it explicit at the API level means the sender takes responsibility for the SLO, and the system can route accordingly.
Idempotency key is required. Notification sends are not safely retryable without dedup. The key makes this the caller’s problem to declare, which is the right contract — the system deduplicates but the caller names what “same notification” means.
Response is “accepted,” not “delivered.” Delivery is async. The status endpoint exists for observability, not for synchronous confirmation. This is the API contract that unlocks the entire async architecture.

4. Core design: the notification pipeline as a priority-routed DAG

This is the question. The naive design is a single queue with workers that send notifications. The real system is a pipeline with multiple stages, each with a different concern.

Three approaches to the pipeline shape:

(a) Single queue. All notifications enter one queue. Workers pull from the queue, look up preferences, template the message, and send via the appropriate channel.

Pros: Simple. One scaling knob (more workers).
Cons: A marketing blast of 10M low-priority emails blocks OTPs behind it. No priority isolation. Head-of-line blocking is catastrophic for critical notifications.

(b) Priority queues. Separate queues per priority level (critical, high, normal, low). Workers drain higher-priority queues first.

Pros: OTPs never wait behind marketing. Isolation is structural.
Cons: Need to re-route based on priority at ingestion. Workers are still shared, so a CPU-hungry channel (email rendering) can still slow down a fast channel (push).

(c) Priority × channel matrix. Separate queues per (priority, channel) pair. Each queue has dedicated workers tuned for that channel’s delivery semantics and rate limits.

Pros: Full isolation. Each queue is scaled, rate-limited, and retried independently. A Twilio rate-limit throttle on SMS doesn’t slow down push sends.
Cons: More queues to manage. Configuration surface grows.

The SDE III answer is: priority × channel matrix, with a shared ingestion stage that routes. The ingestion stage (one queue) handles dedup, preference lookup, and fan-out to per-channel queues. The per-channel queues handle delivery, retry, and channel-specific rate limiting independently.

This separation is the core of the design: a single event enters the system once, is routed to N channels based on preferences, and each channel operates at its own cadence without blocking the others.

Note

Interview signal: The routing/delivery separation is the architectural insight that makes this a strong answer. “Router = who gets what on which channel” and “Delivery worker = sending via this channel’s semantics” are different jobs that scale differently and fail differently. Naming this distinction, and drawing the boundary explicitly, is what most candidates miss.

5. Data model and storage

user_preferences:
  user_id (PK)
  channels_enabled: { push: bool, email: bool, sms: bool, in_app: bool }
  quiet_hours: { start, end, timezone }
  frequency_cap: { channel, max_per_hour }

notification_events:
  notification_id (PK)
  event_type
  user_id
  priority
  data (JSON)
  created_at
  idempotency_key (unique index)

delivery_log:
  notification_id
  channel
  status (pending | sent | delivered | failed | bounced)
  attempt_count
  last_attempt_at
  provider_message_id

Storage choices:

user_preferences — Redis for hot reads (sub-millisecond preference lookups on every notification), backed by DynamoDB for durability. The access pattern is exactly GET preferences WHERE user_id = X at 50K/sec peak — this is what Redis does best. Preferences change rarely (user edits settings), so cache invalidation is simple: write-through on update, TTL for safety.
notification_events — append-only event store. Cassandra — wide partitions keyed by user_id, clustered by created_at. Query pattern is “last N notifications for user X” (for in-app notification center) and “lookup by notification_id” (for status API). Both are Cassandra’s sweet spot.
delivery_log — same shape as notification_events, same store. Append-only, queried by notification_id and time range. Drives the status API and operational dashboards.
Dedup store — Redis with SET NX on the idempotency key, TTL of 24 hours. If the key already exists, the notification is dropped. This is the simplest dedup pattern and works because idempotency keys are caller-defined and short-lived.

I’d say explicitly: “The hot path — preference lookup and dedup check — is entirely in Redis. The durable stores (DynamoDB, Cassandra) are behind it for persistence and the cold reads. Nothing in the critical send path hits a database that isn’t in-memory.”

6. The notification pipeline: ingestion and fan-out

The architecture is a staged pipeline. Each stage has a specific responsibility and scales independently.

flowchart LR
  Producer[Producer service] --> API[Notification API]
  API --> IngestQ[(Ingestion queue · Kafka)]
  IngestQ --> Router[Router workers]
  Router --> PrefCache[(Preferences · Redis)]
  Router --> PushQ[(Push queue)]
  Router --> EmailQ[(Email queue)]
  Router --> SMSQ[(SMS queue)]
  Router --> InAppQ[(In-App queue)]

Stages:

Notification API. Accepts the event, validates, assigns a notification_id, writes to the ingestion queue. Returns immediately. No delivery work happens synchronously.
Ingestion queue. Kafka — durable, ordered per partition, high throughput. Partitioned by user_id so that all notifications for one user are processed in order (important for dedup and frequency caps).
Router workers. Pull from the ingestion queue and do the routing work: check dedup (Redis SET NX), look up user preferences (Redis), check frequency caps, check quiet hours, apply template per channel, and emit one message per enabled channel into the channel-specific queues.

The router is the brains of the system. It makes the send/don’t-send decision per channel per user, respecting all the user state. The channel queues downstream are dumb delivery workers — they just send what they’re told, with retry logic.

7. Channel delivery and rate limiting

Each channel queue has dedicated workers that handle the channel’s specific delivery semantics.

flowchart LR
  Producer[Producer service] --> API[Notification API]
  API --> IngestQ[(Ingestion queue · Kafka)]
  IngestQ --> Router[Router workers]
  Router --> PrefCache[(Preferences · Redis)]
  Router --> PushQ[(Push queue)]
  Router --> EmailQ[(Email queue)]
  Router --> SMSQ[(SMS queue)]
  Router --> InAppQ[(In-App queue)]

  PushQ --> PushWorker[Push workers]
  PushWorker --> APNS[APNs / FCM]

  EmailQ --> EmailWorker[Email workers]
  EmailWorker --> SES[SES / SendGrid]

  SMSQ --> SMSWorker[SMS workers]
  SMSWorker --> Twilio[Twilio]

  InAppQ --> InAppWorker[In-App workers]
  InAppWorker --> WSGateway[WebSocket gateway]

  classDef new fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class PushWorker,EmailWorker,SMSWorker,InAppWorker,APNS,SES,Twilio,WSGateway new

Channel-specific concerns:

Push (APNs/FCM). Token management per device. Token rotation when users reinstall. Batch API for throughput (FCM supports 500 messages per batch request). Feedback loop for invalid tokens.
Email (SES/SendGrid). Rate limits per sender domain. Warm-up for new IP pools. Bounce handling that feeds back into preference store (hard bounce → disable email for that user). HTML rendering per template.
SMS (Twilio). Highest cost per message. Rate-limited by the provider. Reserved for critical notifications (OTPs, security alerts). Delivery receipts are delayed (seconds to minutes).
In-app. Stored in the notification center (a read from Cassandra) and pushed in real-time to connected clients via WebSocket or SSE. No third-party dependency; fastest and cheapest channel.

Rate limiting per channel. Each channel worker implements a token bucket keyed on the provider’s rate limit. When the bucket is empty, the worker pauses (backpressure) rather than dropping messages. The queue absorbs the burst; the worker drains it at the maximum safe rate.

Priority isolation within a channel. Each channel queue is actually two or more sub-queues (critical + normal). Workers always drain the critical sub-queue first. An OTP SMS never waits behind a marketing SMS.

8. Failure modes

I’d proactively walk through what breaks. This is where SDE III differentiates on this question, because notification failures have unusual shapes.

Third-party provider is down (SES, Twilio, APNs). The channel queue backs up. Retry with exponential backoff. The user doesn’t receive on that channel. Mitigation: for critical notifications, the router can fallback to an alternate channel (SMS fails → try push, push fails → try email). Explicitly designing the fallback chain per priority level is the SDE III move.
Kafka is down. Notifications stop flowing. The API can buffer briefly (in-memory with disk spill) or return 503 to callers. This is a total outage of the notification system. Mitigation: Kafka in multi-AZ with replication factor 3. A full Kafka outage is rare but not impossible; the blast radius is “all notifications delayed,” not “all notifications lost” (messages are durable in the producing service’s retry buffer).
Redis (preferences) is down. Router can’t look up preferences. Options: fail open (send on all channels — annoying but safe for critical, wrong for marketing) or fail closed (don’t send — safe for marketing, wrong for OTPs). The right answer depends on priority: critical notifications fail open; non-critical fail closed. Worth saying out loud.

Warning

The preference-race failure mode — user disables email while a notification is already in the email queue — is a trust violation, not just a technical bug. Check preferences again at send time (in the channel worker), not only at routing time. Double-checking is cheap; sending an unwanted email to an opted-out user is a GDPR/CCPA incident in some jurisdictions.

Duplicate event (idempotency key collision failure). If the dedup Redis expires the key (TTL) and the producer retries after the window, the user gets a duplicate. Mitigation: dedup window must be longer than the maximum retry window of any producer. 24 hours is the standard conservative choice.
User sees notification on wrong channel (preference race). User disables email, but a notification was already in the email queue. Mitigation: check preferences at send time (in the channel worker, not just in the router). Double-check is cheap; sending an unwanted email is a trust violation.
Marketing blast overwhelms the system. A campaign targets 50M users simultaneously. The ingestion queue can absorb it, but the channel workers (especially email) can’t drain it fast enough. Mitigation: marketing campaigns are sent at low priority, rate- limited at the router stage (max 100K/min for a single campaign), and never permitted in the critical path. Worth designing a separate “campaign” API that enforces this rather than relying on callers to set priority=low.

The pattern to notice: name what fails, name what degrades gracefully, name what doesn’t. The in-app channel is the most resilient (no third-party). Email/SMS are the most fragile (third-party rate limits). Push sits between.

9. What I’d skip, and say I’m skipping

Time check: five minutes left. Things I’d explicitly defer:

Notification analytics (open rates, CTR). A real system needs this. It’s a downstream consumer of delivery events, not part of the core send path. Worth mentioning, not drawing.
User segmentation engine. The API accepts a segment_id; how that segment is computed (who qualifies) is a separate system — a user-data platform. We assume it’s resolved before the notification API is called.
A/B testing notification content. A real product concern. Orthogonal to the delivery infrastructure. The template layer could support variants, but it’s not the architecture question.
Internationalization and localization. The template engine should support it; the architecture doesn’t change.
Notification grouping and digests. A real and hard problem (batch 8 similar notifications into one digest email). Would need a separate batching stage with a time window. Worth acknowledging the complexity; not designing in this 45 minutes.

Saying “I’d skip this, and here’s why” shows I know the full surface and chose to draw what matters.

10. Wrap-up

One crisp sentence:

This design separates routing (who gets what, on which channel) from delivery (sending it via the channel’s semantics), allowing each channel to operate at its own cadence with its own failure modes, while guaranteeing that critical notifications never wait behind bulk sends.

That framing — priority isolation through structural separation — is the core trade-off and the kind of articulation that lands.

What separates SDE II from SDE III on this question

SDE II draws a queue, names push/email/SMS workers, mentions user preferences, and sketches the happy path.
SDE III separates the routing stage from the delivery stage as a deliberate architectural choice, names priority isolation as the binding constraint (not throughput), designs the fallback chain for critical notifications, walks through the preference-race edge case, and proactively addresses the campaign-blast backpressure problem.
Staff/Principal asks the questions that reveal the hidden product requirements: “What’s the SLA for an OTP notification — if it’s 15 seconds, the architecture is different than if it’s 2 seconds?” They identify systemic risks: a single large marketing campaign can degrade the entire system if campaign routing isn’t rate-limited at ingestion. They think operationally: how does on-call know when notification delivery for a specific channel is degraded, and at what delivery failure rate do we page? They surface missing requirements: compliance with per-country notification laws (e.g., DPDP Act in India, GDPR in Europe) means the “send to all channels on failure” fallback might be illegal in some jurisdictions.

The difference isn’t knowing more channels exist. It’s naming the routing engine as the center of gravity and showing why every other decision (queue topology, rate limiting, fallback chains) falls out of that separation.