Walkthrough: Designing a Recommendation System

The problem

You’re asked to design a recommendation system — YouTube’s “recommended for you,” Amazon’s product suggestions, TikTok’s For You page. The interviewer says something like: “Given a catalog of 100M items and 500M users, show each user items they’re likely to engage with. Recommendations should reflect recent behavior.”

Sounds like an ML question. It mostly isn’t. The trap is spending the interview on model architecture when the system design question is the serving funnel — you cannot score 100M items per request, so the architecture exists to cheaply cut 100M to a thousand (recall), expensively rank the thousand, and feed every click back into features and training. The model is a component; the funnel, the feature pipelines, and the feedback loop are the design.

Below is how I’d walk through this, start to finish.

1. Requirements

First 3–5 minutes. Questions I’d ask to pin down what we’re building:

Functional requirements

• R1. Serve personalized recommendations at request time (home page, item-detail “more like this”).
• R2. Blend multiple candidate sources — collaborative (“users like you watched”), content similarity, trending, fresh items.
• R3. Rank candidates with a learned model over fresh features.
• R4. Feedback loop — impressions and interactions flow back into features and training data.
• R5. Exploration and diversity — surface new/cold items, avoid ten near-identical results.

Scope

Explicitly out of scope:

• Model architectures (two-tower vs transformers — I name what each stage needs from its model, not the layers)
• Ads blending and auctions (a revenue system grafted onto this funnel; its own question)
• Content moderation/integrity (a filter stage I’ll mark in the funnel and not design)
• The A/B experimentation platform (assumed; I’ll name where it hooks in)

Non-functional requirements

• Latency. End-to-end budget for the recommendation call? It gets divided among funnel stages, so it’s the first number I need.
• Freshness, twice. How quickly must behavior show up (you just watched three cooking videos)? How quickly must items show up (uploaded a minute ago)? These stress different components.
• Scale. Requests/sec, corpus size, DAU — the funnel ratios.
• Quality measurement. Is there an online metric (engagement lift) the design should optimize for measurability?

Say the interviewer confirms: 500M DAU, 100M items, 100K requests/sec peak, p99 ≤ 200 ms, behavioral signals fresh within ~1 minute, new items recommendable within ~10 minutes, daily full retrain with continuous fine-tuning, and everything ships behind A/B experiments.

2. Capacity estimate

• Serving: 100K req/sec × ~1K candidates ranked per request = 100M model scores/sec. At even 100 µs per score, that’s 10K CPU-seconds per second — a few thousand cores for ranking alone. Scoring the corpus per request would be 10¹³ scores/sec: five orders of magnitude past possible. Recall isn’t an optimization; it’s what makes the system exist.
• Feature reads: 1K candidates × ~50 feature groups per score → ~5M key-value lookups/sec against the online feature store (batched per request, but the store is still the hottest read system in the design).
• Events: 500M DAU × ~200 impressions + interactions/day ≈ ~1.2M events/sec average into the feedback pipeline — bigger than the serving traffic, which is typical and worth saying.

I’d say out loud: “Two conclusions. The funnel ratio — 100M to 1K to 50 — is the architecture. And the feedback side is bigger than the serving side: more events flow in than requests, so the data pipeline is a peer system, not plumbing.”

3. High-level architecture

Three logical subsystems:

1. Serving funnel — recall sources → feature hydration → ranking → policy/re-rank. Synchronous, latency-budgeted.
2. Feature platform — event ingestion, batch + streaming feature computation, an offline store (training) and an online store (serving).
3. Training loop — dataset assembly from logged impressions, training, evaluation, model registry, rollout.

flowchart TD
  Client[Client] --> Gateway[Rec API]
  Gateway --> Recall[Recall sources]
  Recall --> Rank[Ranker]
  Rank --> Gateway
  Rank --> FS[(Online feature store)]

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class FS store

The feature platform decides the system’s character: serving and training must compute the same features from the same definitions, or the model you trained is not the model you’re serving — the skew problem that quietly ruins these systems.

4. Architecture evolution

Each level keeps the previous as a fallback — and unlike most systems, here that’s load-bearing: when the Level 4 ranker is overloaded, the system degrades down the evolution table (Level 3’s single recall order, Level 1’s trending) rather than failing.

The design in this walkthrough is Level 4.

I’d say out loud: “I’m designing Level 4 — multi-source recall with two-stage ranking. The degradation story is the evolution table read backward, and I’ll make that explicit in failure modes.”

Target architecture (Level 4)

Every section that follows explains one part of this diagram:

flowchart TD
  Client[Client] --> Gateway[Rec API]
  Gateway --> Recall[Recall: ANN + CF + trending + fresh]
  Recall --> ANN[(Embedding index)]
  Recall --> Light[Light ranker]
  Light --> Heavy[Heavy ranker]
  Heavy --> Policy[Re-rank: diversity, explore, filters]
  Policy --> Gateway

  Light --> FS[(Online feature store)]
  Heavy --> FS

  Client -->|events| Log[(Event log)]
  Log --> Stream[Streaming features]
  Stream --> FS
  Log --> Batch[Batch features + training data]
  Batch --> OFS[(Offline feature store)]
  OFS --> Train[Training] --> Registry[(Model registry)]
  Registry -.-> Heavy
  Registry -.-> ANN

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class ANN,FS,Log,OFS,Registry store

5. Core design choice: where the work happens

Three options:

(a) Precompute per user. Nightly batch writes the top-500 list for every user into a KV store; serving is one read. Wonderfully cheap to serve — and stale by up to a day (R4 broken: this morning’s session can’t influence this afternoon), storage scales with users × list length, and a new item can’t appear for anyone until the next batch (R5 broken). This is Level 2, and the right answer for email digests — say so.

(b) Score everything online. Perfect freshness; five orders of magnitude over budget (§2). Dead on arrival, but naming the arithmetic that kills it is what justifies everything after.

(c) Two-stage funnel. Recall stages use precomputed structures (embedding indexes, co-occurrence lists, trending counters) that make “find plausible candidates” an index lookup, not a scoring pass. Ranking then spends the real model only on survivors, with request-time features. Precomputation moves from “the answer” (option a) to “the index that makes the online answer cheap” — that relocation is the design.

Tradeoff matrix

Architecture decisions

Core data structures

6. Read path: the serving funnel [R1, R2, R3, R5]

The slice of the target architecture a request touches, with the budget spent per stage:

flowchart TD
  Client[Client] --> Gateway[Rec API]
  Gateway -->|"~40 ms"| Recall[Recall: 4 sources in parallel]
  Recall --> ANN[(Embedding index)]
  Recall -->|"~5K candidates"| Light[Light ranker · ~20 ms]
  Light -->|"~500"| Heavy[Heavy ranker · ~80 ms]
  Heavy -->|"~500 scored"| Policy[Policy layer · ~10 ms]
  Policy -->|"top 50"| Gateway
  Light --> FS[(Online feature store)]
  Heavy --> FS

  classDef store fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class ANN,FS store

Step by step:

1. Recall, in parallel (R2). Four sources, each returning ~1–2K item IDs: the ANN index queried with the user’s embedding (“items near your taste vector”); item-item co-occurrence seeded by recent interactions (“watched X → watch Y”); trending (time-decayed counters, the cold-start workhorse); and a fresh-item source (recent uploads matched on content features — R5’s supply side). Union and dedup to ~5K. Sources are deliberately overlapping and individually mediocre — recall’s job is coverage, not precision; precision is the next stage’s job.
2. Light ranking. A cheap model (GBDT or a small NN) over a handful of features scores all 5K in ~20 ms and keeps 500. Its only job is to not drop gems — it’s tuned for recall of the heavy ranker’s eventual winners.
3. Feature hydration + heavy ranking (R3). Batched reads from the online feature store hydrate ~50 feature groups for 500 candidates; the full model scores p(engage) per item. This is the stage that knows about this user, right now — session features computed from the request itself (last 10 minutes of behavior) ride alongside stored ones.
4. Policy re-rank (R5). Deterministic, debuggable rules over scores: diversity (cap per category/author via MMR-style penalty), exploration (an ε slice of slots goes to high-uncertainty items — the feedback loop’s oxygen), business filters (already-seen suppression, integrity filters).
5. Log the impression. The response’s item list, the model version, and the features as seen at scoring time are logged with a tracking token — this record is tomorrow’s training data (§8), and writing it here, not reconstructing it later, is what makes training honest.

7. Feature platform [R3, R4]

The dual-store layout, and the one correctness rule that governs it:

Online store — a KV system serving (entity, feature_group) rows at ~5M lookups/sec, single-digit-ms, holding the current value of every feature: user long-term preferences (batch-computed nightly), item statistics (streaming, minutes fresh), user recent-behavior aggregates (streaming: “videos watched last hour”, updated from the event stream within ~1 minute — R4’s freshness requirement lands here).

Offline store — the same features, with history, in the warehouse: every value with its valid-time range, so training can ask “what did this feature equal at the moment of this impression?”

The rule: one definition, two materializations. Each feature is defined once (a transformation over the event log); a batch job materializes it into the offline store and a streaming job into the online store. The moment a feature is computed one way for serving and another for training — training-serving skew — the model learns patterns that don’t exist at serving time, and quality decays silently: no errors, no alerts, just a model quietly wrong about the world it sees.

Point-in-time correctness. Assembling training data joins impressions to feature history: the example for Tuesday 9:14’s impression must contain the features as of Tuesday 9:14 — not today’s values. Joining to current values (label leakage) trains a model on information from the future; it will look brilliant offline and underperform live, and the gap will read as “the model is fine, serving must be broken.” Logging features at scoring time (§6 step 5) sidesteps the join for everything the ranker saw — the strongest practical defense, at the cost of log volume.

8. Training loop [R4]

1. Dataset assembly. Join the impression log (what was shown, features at scoring time) with the event log (what happened next) under an attribution window: impression + click/watch → positive; impression + no engagement → negative. The subtle half: unshown items generate no labels at all — the system only learns about what it chose to show (§11’s feedback loop).
2. Train the heavy ranker daily on the trailing window; fine-tune more frequently on recent data. The two-tower embedding model retrains on the same cadence; its item tower re-embeds the corpus and rebuilds the ANN index (new items enter the index incrementally between rebuilds via the content-feature path — the ~10-minute new-item SLO from §1).
3. Evaluate and gate. Offline metrics (AUC, recall@K against held-out interactions) gate promotion into the registry; online, every model ships as an A/B arm. The honest framing to say aloud: offline metrics are a filter, not a verdict — the offline-online gap is structural (offline data only covers shown items), which is why the experiment platform is load-bearing rather than nice-to-have.
4. Roll out via the registry: ranker instances load the new version, shadow-score for a warmup window, then take traffic. Rollback is a registry pointer flip — model deploys are the system’s most frequent change, so they get the cheapest undo.

9. API design [R1, R4]

GET /v1/recommendations?surface=home&count=50          [R1]
  headers: user auth; body/params: session context (device, locale)
  returns: { items: [{item_id, tracking_token}, ...], request_id }

POST /v1/events                                        [R4]
  body: [{ tracking_token, event: impression|click|watch|skip,
           dwell_ms?, position }]

Two non-obvious decisions:

Every recommended item carries a tracking token. The token encodes (request_id, model_version, position, feature-log pointer). Events posted back with the token close the loop exactly: this click belongs to that impression, scored by that model, seeing those features. Without it, attribution degenerates into timestamp-joining heuristics — and the training data inherits the fuzz.

Pagination is a session cursor, not a re-rank. Page 2 must come from the page-1 funnel result (cached by request_id), not a fresh funnel run — fresh runs reshuffle, duplicate page-1 items, and burn 10× compute on scrollers. The cost: results within a session are deliberately stale by design, which is correct UX anyway.

The API hides every stage of the funnel — surfaces, sources, models can all change behind GET /recommendations. What it refuses to hide is attribution: the token is the feedback loop’s contract with the client.

10. Data model and storage

Requirements → components

Access frequency

Storage technology choices

Access pattern matrix

Schemas

Impression log [R4] — the system’s most valuable table:

impression_log (columnar, partitioned by hour):
  request_id, user_id, surface
  model_version, position, item_id
  features_snapshot:   the feature vector as scored   # anti-skew, anti-leakage
  policy_flags:        explored? diversity-boosted?

Online feature rows [R3]

online_features (KV):
  key:    (entity_id, feature_group)        # user:123, "engagement_1h"
  value:  packed floats + updated_at
  ttl:    per group — session features minutes, profile features days

11. Failure modes

Grouped by what they threaten. The pattern: name what fails, name what degrades, name what doesn’t.

Latency failures

Heavy ranker overload

Fails: the 80 ms ranking stage under a traffic spike or a slow new model. Degrades — by design, down the evolution table: shrink heavy-stage width (500 → 150), then skip to light-ranker order, then serve recall blend with trending. Each rung is a measurable quality drop and a measured latency win; the ladder is pre-built and load-tested, not improvised. Doesn’t degrade: availability — a popularity-ordered response is a worse page, never an error page.

Online feature store partial outage

Fails: some feature groups’ reads. Degrades: rankers score with default values for missing features — which only works if the model trained with feature dropout, so defaults are in-distribution; this is a training-time decision made for a serving-time failure, worth saying out loud. Doesn’t degrade: the funnel shape itself; quality dips, traffic flows.

Freshness failures

Streaming feature pipeline lags

Fails: the ~1-minute behavioral freshness — “just watched three cooking videos” stops steering. Degrades: rankers fall back to batch-computed profile features; recommendations get yesterday-shaped, not wrong. The session-feature path (computed from the request itself) survives independently — a deliberate redundancy. Doesn’t degrade: serving latency or coverage.

Embedding index lag

Fails: new items’ visibility in ANN recall (index rebuild stalls). Degrades: the fresh-item recall source (content-feature matching, no embedding needed) carries cold-start alone — R5’s supply path is the backup for R2’s main path. Doesn’t degrade: recommendations for established items.

Correctness failures (the quiet ones)

Feedback loop bias

Fails: the training distribution, structurally — the model learns only from what previous models chose to show, so popular items gather more data, rank higher, gather more data. Position bias compounds it (slot 1 gets clicked because it’s slot 1). Degrades: catalog diversity and new-item success rate, over weeks, with every dashboard green. Mitigations are policy-level: the exploration slice (§6 step 4) generates off-policy data on purpose; logged position feeds the model as a feature (or inverse-propensity weighting in training); diversity floors in re-rank. Doesn’t degrade: any metric anyone is paged on — which is exactly why it must be designed against, not detected.

Training-serving skew

Fails: the equivalence between trained and served models (§7). Degrades: online quality, silently, while offline metrics stay excellent — the signature is the offline-online gap widening release over release. Mitigations: scoring-time feature logging, nightly recompute-and-diff audits, schema versioning between registry and feature store. Doesn’t degrade: anything visible on day one — this is the failure that costs a quarter if unwatched.

12. What I’d skip, and say I’m skipping

Time check — five minutes left. Things I’d explicitly defer:

• Model internals. Two-tower, GBDTs, sequence models — I’ve specified each stage’s contract (cost, input features, output); architecture choice inside the contract is the ML team’s iteration loop, deliberately decoupled by the registry.
• The experimentation platform. Every model and policy change ships as an experiment; designing the assignment/metrics system is its own question.
• Ads. Blending paid items is an auction grafted into the policy layer with its own incentives — a different interview.
• Integrity/moderation filters. A filter stage in recall and policy; consuming its verdicts is easy, producing them is not this system.
• Cross-surface coordination. Home page vs notifications vs email competing for the same user attention — real, fascinating, out of scope.

Saying “I’d skip this, and here’s why” is a strong signal. It shows you know the full surface and are making deliberate scoping choices.

Interview flow summary

Architecture at a glance

SERVING FUNNEL (200 ms)                 FEEDBACK LOOP (continuous)

Request                                 Impressions + events
 ↓                                       ↓
Recall ×4 sources  → ~5K                Event log (replayable)
 ↓ light ranker    → ~500                ↓
Feature hydration (online store)        Streaming features → online store (~1 min)
 ↓ heavy ranker    → scored             Batch features → offline store (history)
 ↓ policy: diversity, explore            ↓
Top 50 + tracking tokens                Point-in-time training data
 ↓                                       ↓
Log impressions + feature snapshots     Train → evaluate → registry → rollout

The walkthrough order for your whiteboard

1. Requirements        — two freshness clocks; the latency budget
2. Capacity            — the funnel ratio; feedback bigger than serving
3. Evolution           — popularity → precompute → two-stage → multi-source
4. Core choice         — precompute vs online vs funnel; the arithmetic
5. Serving funnel      — recall union, light/heavy split, policy layer
6. Feature platform    — dual store, one definition, skew and leakage
7. Training loop       — impression joins, gates, registry rollout
8. API                 — tracking tokens, session-cursor pagination
9. Data model          — impression log with feature snapshots
10. Failure modes      — degradation ladder, then the two silent ones

If the interviewer pushes deeper

13. Wrap-up

One crisp sentence before the interviewer’s next question:

This design spends a fixed latency budget down a funnel — cheap indexes cut 100M items to thousands, a real model ranks hundreds with minute-fresh features — and treats the feedback loop as the second product: impressions are logged with the exact features that produced them, so the system that learns is provably the system that served.

What separates levels on this question

• SDE II names collaborative filtering and “an ML model ranks the results,” sketches one recall source and one ranker, and treats features as “data the model uses.”
• SDE III derives the funnel from the latency arithmetic, splits ranking into light and heavy stages with budgets, designs the dual feature store with streaming freshness, closes the loop with tracking tokens and impression logs, and walks the degradation ladder unprompted.
• Staff/Principal treats the quiet failures as the design’s center — names training-serving skew and label leakage with their defenses, frames exploration as paying data-debt rather than losing clicks, questions the metric itself (engagement vs satisfaction), and prices the system honestly (feature-store reads and ranker cores per request — and which funnel width to cut when finance asks).

The difference isn’t knowledge. It’s which system you think you’re designing — weaker answers design the model’s surroundings; stronger ones design the loop, knowing the model is a replaceable tenant inside it.

Further reading

• Deep Neural Networks for YouTube Recommendations — Covington et al. The paper that canonized the two-stage (candidate generation + ranking) architecture this walkthrough builds on.
• Designing Data-Intensive Applications — Kleppmann. Chapters 10–11 (batch and stream processing) are the theory under the feature platform’s dual materialization.
• System Design Interview Vol. 1 — Alex Xu. Chapter 11 (news feed) is the adjacent question; this design is what its “ranking” box expands into.
• Meet Michelangelo: Uber’s Machine Learning Platform — the canonical feature-store writeup: one definition, online/offline materialization, and why the platform exists.
• System Design for Recommendations and Search — Eugene Yan. Practitioner patterns for the retrieval/ranking split across real companies’ published architectures.
• Walkthrough: Designing a Vector Database — the ANN index that recall queries, designed in full.