You are a strong backend system design interviewer. You’ve designed URL shorteners, rate limiters, news feeds. You can talk consistent hashing in your sleep. You’re now in an AI/ML system design loop and you can tell the interviewer is unsatisfied with answers that worked in every backend loop you’ve done.
The standard advice — “study ML system design” — is unhelpful, because it implies the entire interview is different. Most of it isn’t. The frame that actually helps is sorting which parts of your existing skill apply unchanged, which apply with a translation, and which are genuinely new content you have to build from scratch.
That sorting is what this framework is for.
The problem
The naive paths through this interview both fail.
Path A: treat it like a backend interview. You design a request flow, sketch a database, talk caching and sharding, name failure modes. The interviewer nods politely and asks a follow-up question that’s clearly about ranking quality, drift, or eval. You answer with infrastructure language. The signal lands as “strong backend, weak ML.”
Path B: treat it like a research interview. You over-index on model selection, hyperparameter tuning, training pipeline. You talk about embeddings as if the architecture’s job is to produce them. You under-spec the parts of the system that look like normal serving. The signal lands as “knows ML buzzwords, can’t ship a system.”
The interviewer is looking for neither extreme. They’re looking for someone who can integrate: who treats the ML components as first-class infrastructure with their own failure modes and SLOs, while still designing a real system around them. The framework below is how to identify which gear to be in for each part of the question.
The framework: three layers, three questions
Every AI/ML system design question has three layers stacked on top of each other. Most candidates collapse them or skip one.
| Layer | What it is | Looks like backend? |
|---|---|---|
| Serving | The request path. Auth, routing, batching, caching, latency. | Yes — backend skill applies directly |
| ML core | The model(s), retrieval, ranking, embeddings, eval. | Partially — backend translates, but new content |
| Lifecycle | Training/finetuning, data pipelines, eval loops, monitoring. | No — this is mostly new |
The three questions, applied to each layer:
- What does this layer optimize for?
- What is its SLO and how is it measured?
- What is its dominant failure mode?
The framework’s claim: a strong AI/ML system design answer covers all three layers, and answers the three questions for each. A weak answer treats the ML core as a single black box (which the interviewer reads as “doesn’t know what’s inside”) or treats lifecycle as out of scope (which the interviewer reads as “doesn’t know real ML systems break in production for offline reasons”).
Layer 1: Serving — backend skill applies directly
This is the layer you already know. Your existing backend system design skill covers it without translation. Specifically:
- Request routing, gateways, auth, rate limiting. Same as any backend system. An LLM serving platform’s gateway looks like any API gateway.
- Sharding, replication, autoscaling. Same primitives. The constrained resource is GPU instead of CPU/memory, but the principles transfer.
- Caching at the request level. Embedding caches, response caches, prefix caches — these are caches with TTLs and invalidation semantics, same as any cache, just with ML-specific access patterns.
- Async pipelines. Click events, eval traces, training data collection — these are Kafka pipelines like any other.
What’s workload-specific but still backend-shaped: continuous batching in inference (the request lifecycle is unusual but the scheduler is just a scheduler), KV cache memory management (resource accounting with weird shapes), prefix-cache-aware routing (consistent hashing with a workload twist).
What you optimize for at this layer: throughput per dollar, p95 latency, multi-tenancy fairness. Standard.
What the SLO looks like: TTFT (time-to-first-token) p50/p95, ITL (inter-token latency), tokens/sec/GPU. The unit metrics differ from a typical backend service, but they’re still latency and throughput.
Dominant failure modes: GPU OOM, cold-start asymmetry (model weights take minutes to load), request cancellation. See the LLM serving walkthrough for these in depth.
If you’re a strong backend candidate, you should expect to spend ~30% of the interview here and feel comfortable the entire time. The mistake is either skipping this layer because it’s “obvious” — leaving the interviewer to guess whether you can actually design serving — or spending 70% of the interview here because it’s where you’re confident.
Layer 2: ML core — translates from backend, but new content
This is where strong backend candidates leak the most signal. The ML core is genuinely different content, but it’s backend-shaped content if you frame it right.
The translation table:
| Backend concept | ML core analog |
|---|---|
| Database query plan | Retrieval pipeline (BM25 + vector + rerank) |
| Index choice (B-tree vs hash) | ANN index choice (HNSW vs IVF vs Flat) |
| Schema design | Feature store schema, embedding dimensions, chunking strategy |
| Cache hit rate | Retrieval recall@k |
| Query latency | Retrieval latency budget within end-to-end budget |
| Read consistency model | Eval consistency (offline ↔ online metric drift) |
| Idempotency | Determinism / sampling settings |
The point of the table isn’t that the analogs are exact — they aren’t. It’s that you can use your existing instincts to ask the right questions. A backend engineer who’d ask “what’s the access pattern on this table?” should ask “what’s the query distribution against this retrieval index?” and get the same architectural value.
Specific things in this layer that need their own content:
- Retrieval as a discipline. BM25, embedding-based dense retrieval, hybrid retrieval, reranking. This is 20-year-old IR vocabulary, mostly stable. The RAG walkthrough is a worked example.
- Ranking. Multi-stage funnels (retrieve → rerank → optionally re-rank with a cross-encoder), feature engineering for rankers, the candidate generation/ranking split.
- Embeddings. What an embedding is, how to choose model and dimensionality, normalization, the cost of re-embedding when the model changes.
- Eval. Offline metrics (recall, MRR, nDCG) vs online metrics (CTR, task completion). Why they drift. Why this drift is the dominant silent failure mode.
What you optimize for at this layer: quality (recall, ranking quality, generation faithfulness), measured against an eval set, with quality/cost as the operative trade-off.
What the SLO looks like: retrieval recall@k, ranking nDCG@k, generation faithfulness rate, hallucination rate. These are quality SLOs, not latency SLOs. A system that returns the wrong answer in 100ms is worse than one that returns the right answer in 500ms.
Dominant failure modes: silent quality degradation (the system stays up, the metrics don’t), retrieval recall drops as the corpus grows, embedding model swaps invalidate the index. The signature insight to volunteer is that the ML core’s failure modes are mostly silent — they don’t page someone, they just make the product worse.
A useful test for whether you’re handling this layer well: can you state the eval methodology before designing the ML core? If you can’t say “I’d build a 500-query eval set with relevance labels and measure recall@10 before swapping anything in this pipeline,” you’re guessing about quality the same way the candidates the interviewer doesn’t hire are guessing.
Layer 3: Lifecycle — mostly new content
This is the layer most backend candidates skip entirely, and it’s the single biggest differentiator. ML systems break in production for offline reasons more often than for serving reasons.
What lives here:
- Training and fine-tuning pipelines. Data ingestion, labeling, cleaning, training, validation. Not always relevant if you’re using a hosted model — but always relevant for retrieval index refreshes, embedding regeneration, ranker retraining.
- Eval pipelines. Offline eval (run held-out queries, measure metrics), shadow eval (run new model on prod traffic, compare to current), online eval (A/B with confidence intervals, guardrail metrics).
- Data drift and model drift detection. Distribution monitors. When to retrain. How to know the eval set has gone stale.
- Feedback loops. User clicks, ratings, corrections feeding back into training data. The classic feedback-loop hazard where the model’s outputs become its own training data and bias compounds.
This is where you’ll feel the gap most acutely if you’ve never built it. The good news: you don’t need to know how to do every part — you need to know it exists, what role it plays, and what fails when you skip it.
What you optimize for at this layer: durability of quality over time. Fresh data, refreshed indexes, unbiased feedback, evaluable progress.
What the SLO looks like: eval pipeline runs daily, offline-online metric correlation > some threshold, model retrain cadence vs corpus growth rate.
Dominant failure modes: model drift (the world changed and the model didn’t), feedback-loop bias (system outputs poison its own training set), eval set staleness (the metric you’re optimizing no longer represents the user). All silent. All deadly. All the kind of thing that distinguishes someone who’s only read about ML systems from someone who’s owned one.
The signal-worthy line to volunteer: “the eval pipeline is more load-bearing than the serving stack, because the serving stack pages when it breaks and the eval pipeline doesn’t.”
How to apply it: three worked examples
Example 1: Design YouTube search
A backend engineer’s instinct: this is search infrastructure, an index behind a query API. True, but incomplete.
The three-layer breakdown:
- Serving layer: the query API, the autocomplete service, latency budgets, sharding the index. ~25% of the answer. Pure backend.
- ML core: the retrieval funnel (lexical match → embedding match → reranker), the multi-modal piece (video embeddings), how the ranker is trained and what features it uses (watch time, freshness, engagement signals, query-video relevance). ~50% of the answer. Backend instincts apply, but specific ML content lives here.
- Lifecycle: how the ranker is retrained (offline batch on yesterday’s logs), how feedback loops are managed (clicked-but-low-watch is a different signal than not-clicked), how the eval set is maintained (held-out from human raters? from prod logs?). ~25% of the answer. Almost entirely new content for a pure-backend candidate.
The candidate who answers all three layers — and explicitly says “let me also walk through how this thing stays good over time, since search quality decays” — is reading as ML-aware. The candidate who designs the serving and stops is reading as backend-only.
Example 2: Design a fraud detection system
This is the canonical “backend candidates underperform” question, because the ML core is non-obvious.
The three-layer breakdown:
- Serving layer: real-time scoring API at the payment edge, async scoring for batch flags, latency budget (must score within the authorization window), feature store reads. ~30% of the answer.
- ML core: what model class (gradient-boosted trees usually, not deep learning — and the candidate who knows why is signaling), what features (velocity features, graph features, behavior features), how thresholds are set (precision/recall trade-off varies by transaction size), and the adversarial nature of the problem (the data distribution changes when you ship the model). ~40% of the answer.
- Lifecycle: retrain cadence (weekly, with rapid hotfix capability), eval methodology (you can’t A/B because letting fraud through is expensive — shadow mode is the answer), label feedback (chargebacks arrive 30+ days late, which means your training data is always 30 days old). ~30% of the answer.
The lifecycle layer here is the differentiator. A candidate who says “oh, and we’d retrain weekly” without addressing the chargeback delay, shadow eval, or adversarial drift is missing the question. A candidate who volunteers all three reads as someone who’s actually worked on this class of system.
Example 3: Design an LLM-powered code assistant
This is the modern question. Most candidates think it’s pure serving.
The three-layer breakdown:
- Serving layer: the request flow, streaming SSE, prefix caching (huge here — system prompts and codebase context are highly cacheable), per-tenant routing, rate limiting. ~40% of the answer.
- ML core: model selection (one model? a router that picks between small and large?), context construction (retrieval over the codebase, recent file edits, conversation history), prompt construction patterns, output parsing for tool calls. ~30% of the answer.
- Lifecycle: eval methodology (this is where most teams flounder — what’s a good code completion?), feedback collection (accept/reject, edit-after-accept), model swap discipline (when to swap models, how to measure regression). ~30% of the answer.
The interesting twist for LLM-powered products is that lifecycle isn’t about retraining — you’re not retraining the model. It’s about prompt versions and eval discipline. The lifecycle layer becomes prompt-engineering-as-software-engineering: prompt versions, prompt eval suites, A/B testing of prompts, regression detection. A candidate who says “we don’t have a lifecycle layer because we’re using a hosted model” is signaling they don’t know that prompts are the new code.
Where this framework breaks
Three places.
Pure ML-research-shaped questions. “Design a recommendation system that beats a baseline by X%” — here the question is nearly all ML core, the serving layer is implied, and there’s no system to design in the backend sense. The framework’s three layers still exist, but the weights are skewed enough that the framework gives less leverage. Use an ML-system-design framework instead (the ML System Design Framework piece is the right reference, when written).
Pure infrastructure questions. “Design vLLM” or “Design a vector database” — here the ML core is so embedded in the infrastructure that the layers collapse. Treat these as backend system design questions with ML domain knowledge layered in, rather than as ML system design with infrastructure underneath.
Behavioral-leaning questions. “Tell me how you’d build a team to work on this” — that’s a leadership question wearing a system-design mask. Different framework entirely.
The framework also has a real failure mode worth naming: it implies the three layers are equally weighted, and they aren’t. Different questions weight them differently (see the worked examples). The fix is to ask up front, in your clarifying questions: “is the bulk of this question about how the system is served, how the ML core works, or how it stays good over time?” The interviewer’s answer tells you how to allocate your minutes.
What this framework is really saying
Back up to the meta-position: the AI/ML system design interview is not a different interview. It’s the same interview with two extra layers stacked on top, and most backend candidates underperform because they either skip the new layers or treat them as one big black box.
The fix isn’t to become an ML engineer. It’s to extend your existing system design vocabulary across all three layers — to treat retrieval, ranking, embeddings, and eval as architectural primitives with their own access patterns, SLOs, and failure modes, the same way you treat caches and queues. Then design the integrated system end to end.
That’s the answer interviewers are looking for. It’s also, conveniently, the same skill that lets you actually own ML systems in production — which is the long game this prep is in service of.
Related
- Walkthrough: Designing a RAG System — the canonical AI system design walkthrough, modeling the three-layer approach in practice.
- Walkthrough: Designing an LLM Serving Platform — the serving layer in depth.
- Framework: What’s Durable vs What’s Hype — what to invest in beyond interview prep.
- Plan: Transition to ML Engineer — the structured backend → MLE ladder.
Date-stamped: written 2026-05-26. The three-layer framework structure should hold; specific examples (LLM coding assistants, current model options) will need refresh annually.