What it is
Ranking millions of items for every request with a rich model is impossible: the model costs too much per item. Search and recommendation systems solve it with a funnel: a cheap, recall-oriented stage narrows millions of candidates to thousands, a cheap ranker narrows those to hundreds, and an expensive model scores only that small set. Each stage trades coverage for precision and cost, and each has a latency budget.
Important
Key takeaway: Retrieval is for recall; ranking is for precision. Make the early stages cheap and generous (never drop a plausible item), spend the model budget only on the survivors, and apply deterministic policy (diversity, exploration, filters) last. The ranker is where the product lives; retrieval is plumbing.
When you care
“Design a recommendation system”, “design search ranking”, and “design a feed ranker” all use this shape. The follow-ups are the latency budget per stage, how features are served, how to handle new items and users, and how training data is logged.
Note
Interview signal: Giving the funnel with numbers (about 5,000 candidates, then 500, then the top 50, with a millisecond budget per stage) shows you have reasoned about cost. Saying the early stages are deliberately overlapping and individually mediocre shows you understand their job.
The funnel
| Stage | Input to output | Cost per item | Job | Example latency |
|---|---|---|---|---|
| Retrieval | Millions to about 5,000 | Very low (index lookups) | Do not miss plausible candidates | About 40 ms, sources in parallel |
| Light ranker | 5,000 to about 500 | Low (a small model or a dot product) | Do not drop the heavy ranker’s eventual winners | About 20 ms |
| Heavy ranker | 500 scored | High (a full model, hundreds of features) | Order by predicted engagement or relevance | About 80 ms |
| Policy layer | 500 to the top 50 | Trivial rules | Diversity, exploration, business filters | About 10 ms |
The stage latencies add up to a request budget (here about 150 ms, plus network).
Stage 1: retrieval. Several sources run in parallel, each returning 1,000 to 2,000 ids, then are unioned and deduplicated:
- Embedding nearest neighbors: items near the user’s taste vector in an ANN index.
- Item-to-item co-occurrence: “watched X, so watch Y”, seeded by recent interactions.
- Trending: time-decayed counters, the cold-start workhorse.
- Fresh items: recent uploads matched on content features, the supply side for new content.
- Keyword retrieval for search: the inverted index returns the top candidates by BM25 and a recency boost.
Stage 2: light ranking. A gradient-boosted tree, a small neural network, or a two-tower model scores everything cheaply. In a two-tower model the user and the item each have a learned embedding, and the score is their dot product: one multiply per candidate, and item embeddings can be precomputed. Its only job is recall of the heavy ranker’s winners.
Stage 3: heavy ranking. A batched read from the online feature store hydrates perhaps 50 feature groups for 500 candidates, and the full model predicts something like the probability of engagement per item. Features fall into groups:
| Group | Examples |
|---|---|
| Social affinity | Is the author a friend, a friend of a friend, recently interacted with |
| Engagement | Likes, comments, shares, recent velocity |
| Query or context match | BM25, title match, hashtag match |
| Content | Media type, language, length |
| User and session | Region, language, the last ten minutes of behavior (computed from the request itself) |
Stage 4: policy. Deterministic, debuggable rules over the scores: diversity (cap per category or author with an MMR-style penalty), exploration (a small slice of slots goes to high-uncertainty items, which gives the feedback loop new data), and business filters (already-seen suppression, integrity filters).
Closing the loop: logging and training
Log the impression with the features exactly as seen at scoring time, the model version, and a tracking token. That record is tomorrow’s training data, and writing it at serving time is what makes training honest. If you reconstruct features later, you train on information the model did not have, called training-serving skew, and offline metrics stop predicting online behavior.
Caution
Common mistake: Training on features recomputed after the fact, or on data the model’s own earlier ranking selected without any exploration. The first inflates offline metrics; the second creates a feedback loop where the model only ever learns about what it already showed. Log features at serving time, and keep an exploration slice.
Cold start
New users have no history; new items have no engagement. The funnel handles both with sources that do not depend on history: trending and popularity for new users, content-based retrieval and a fresh-item source for new items, plus exploration slots so new items can earn their first signals.
Warning
Production reality: Each stage has its own failure mode and fallback. If the heavy ranker times out, serve the light ranker’s order; if the feature store is slow, score with the features you have and degrade gracefully. A recommendation page that is slightly worse beats an error page.
When to pick what
| Situation | Choose | Why |
|---|---|---|
| Millions of candidates | A multi-source retrieval stage | Recall at low cost per item |
| Cheap scoring of thousands | Two-tower dot product or a small tree model | One multiply per candidate |
| Rich personalization on a few hundred | A heavy model with an online feature store | Spend the budget where it counts |
| New items or users | History-free sources plus exploration slots | Cold start |
| Diversity and business rules | A deterministic policy layer | Debuggable, controllable |
| Training data | Log features at serving time | Avoids training-serving skew |
Related
- Walkthrough: Designing a Recommendation System — the funnel end to end for a recommendation product.
- Walkthrough: Designing a Full Ranking System — the ranking stages and their latency budgets in depth.
- Reference: Approximate Nearest Neighbor Indexes — the embedding retrieval source at the top of the funnel.