Walkthrough: Designing an LLM Serving Platform

The problem

You’re asked to design an LLM serving platform — the system that sits behind an internal POST /v1/chat/completions endpoint and turns prompts into tokens. The interviewer says something like: “Design a service that hosts large language models, handles thousands of concurrent users, and streams completions back with low latency.”

Sounds like “wrap a model in a Flask app.” It isn’t. This is the canonical GPU-serving question, and what makes it non-trivial is that LLM inference breaks every assumption from a normal serving stack. Requests have unbounded, variable output length. The most expensive resource — GPU memory — is consumed not by the model weights but by a per-request data structure (the KV cache) that grows monotonically as tokens are generated. And throughput vs latency is not a tuning knob; it’s a fundamental tension that the batching strategy either resolves or doesn’t. The interviewer is watching for whether you treat this as a serving problem or a memory-management problem.

Important

The central tension in LLM serving is not request routing — it’s GPU memory management. KV cache for in-flight requests dominates the memory budget, not model weights. Every batching and scheduling decision is downstream of this constraint. Candidates who don’t name this early tend to design for throughput without a mechanism to actually achieve it.

Below is how I’d walk through this, roughly in the order I’d speak the words.

1. Clarify before you design

First 3–5 minutes. Resist the urge to start drawing.

Questions I’d ask:

Which models, and how many. One 70B model, or a fleet of mixed sizes? Multi-model changes the entire scheduler and routing layer.
Request shape. Chat completions, embeddings, or both? Embeddings batch trivially and look almost like normal serving. Chat completions are the hard case.
Latency requirements. What’s the SLO on time-to-first-token (TTFT) and inter-token latency (ITL)? Both matter, and they trade off differently against throughput.
Streaming or non-streaming. Streaming SSE responses change the protocol and the cancellation story.
Throughput target. Tokens per second across the fleet. This sizes GPU count more than QPS does.
Multi-tenancy. Is this for one product or many? Tenant isolation, fairness, and rate limiting all live downstream of this answer.
Hosted or self-hosted. Are we serving open-weight models on our own GPUs, or proxying to OpenAI/Anthropic? Different system entirely.
Hardware. H100s, A100s, or a mix? Drives the batching math.

Say the interviewer confirms: self-hosted on H100s, three models in the fleet (a 7B, a 13B, and a 70B), chat completions with streaming, p50 TTFT < 500ms, p95 < 1.5s, multi-tenant across internal product teams, 100K active users at peak.

2. Capacity estimate

Brief. The point is to check that you understand the dominant cost.

Assume 10K concurrent in-flight requests at peak (100K active users, 10% actively generating at any moment).
Average prompt length 1K tokens, average output length 500 tokens.
A single H100 with a 70B model in fp16 fits the weights (~140GB) only with tensor parallelism across 2 GPUs. With paged KV cache, call it ~30 concurrent requests per 2-GPU shard at decent throughput.
10K concurrent requests / 30 per shard = ~333 shards = ~666 H100s for the 70B model alone. The 7B and 13B add maybe another 50–100 GPUs combined.

I’d say out loud: “This tells me three things. One — GPU memory, not compute, is the binding constraint, because the KV cache for 10K concurrent requests dwarfs the weights. Two — the 70B model dominates capacity planning. Three — batching efficiency directly translates to dollars; a 2x improvement in effective batch size halves the GPU bill.”

3. API design

Two endpoints carry the load:

POST /v1/chat/completions
  body:    { model, messages, max_tokens, stream, temperature, ... }
  returns: SSE stream of token deltas, or single JSON if stream=false

POST /v1/embeddings
  body:    { model, input: string | string[] }
  returns: { data: [{ embedding: float[], index }] }

Decisions worth saying out loud:

Streaming via SSE, not WebSockets. The traffic is one-way server-to-client; SSE is simpler, works through HTTP/1.1 and HTTP/2, and survives proxies cleanly. WebSockets add bidirectional capability we don’t need.
Idempotency. LLM calls are non-idempotent by default (sampling is random). Accept an optional idempotency_key header that the gateway uses for response caching — important because retries on a long-running generation are otherwise extremely expensive.
Cancellation. When the client disconnects mid-stream, the inference worker must stop generating. This is one of the failure modes most teams get wrong. We’ll come back to it.
max_tokens is required. Unbounded generation is a denial-of-service vector. The gateway rejects requests without it.

4. Core design: batching strategy

This is the question. It’s where SDE II and SDE III answers diverge most.

LLM inference has two phases per request: a prefill phase that processes the entire prompt in parallel (compute-bound), and a decode phase that generates one token at a time, autoregressively (memory bandwidth-bound). The batching strategy has to handle both, and they have opposite characteristics.

Three approaches to consider:

(a) No batching. One request per GPU forward pass.

Pros: Simplest. Lowest TTFT.
Cons: GPU utilization is ~5% during decode. Throughput is catastrophic. This is what every “wrap the model in a Flask app” tutorial does, and it is never the answer in production.

(b) Static batching. Collect N requests, run them through prefill and decode together, return all responses when the longest one finishes.

Pros: Higher throughput than no batching.
Cons: Head-of-line blocking. A 50-token request and a 2000-token request in the same batch both wait for the 2000-token one. TTFT is unpredictable because the batch waits for a full window.

(c) Continuous batching. At every decode step, the scheduler decides which requests are in the active batch. Finished requests leave the batch immediately. New requests join on the next step (with a prefill pass that’s interleaved into the decode loop).

Pros: GPU stays near 100% utilization. Short requests don’t wait for long ones. TTFT and throughput both improve.
Cons: Complex scheduler. Requires paged KV cache to handle requests of different lengths efficiently in the same batch.

The SDE III answer is: continuous batching with paged KV cache, following the vLLM design. This is a reference implementation now — SGLang, TensorRT-LLM, and Hugging Face TGI all implement variants. I wouldn’t reinvent the scheduler for an interview; I’d cite the design and explain it.

Note

The interview signal is not whether the candidate knows “continuous batching” by name — it’s whether they explain why static batching fails (head-of-line blocking on the longest request) and why paged attention is the enabling primitive (on-demand block allocation prevents worst-case reservation). Knowing the buzzword without the mechanism reads as surface familiarity, not depth.

5. Algorithm deep-dive: continuous batching with paged attention

Worth walking through, because the interviewer is checking whether you actually understand what makes this work.

Each request occupies KV cache memory proportional to its current token count. In naive serving, you allocate the worst-case KV cache (for max_tokens) up front per request. That wastes most of it — the average request finishes well below max_tokens. With 30 concurrent requests allocating worst-case cache each, you get maybe 10 actually-running requests worth of throughput.

Paged attention fixes this by paging the KV cache. Each request’s cache is broken into fixed-size blocks (typically 16 tokens each), allocated lazily as the request generates. Blocks are stored in a global pool and addressed by a per-request page table. When a request finishes, its blocks return to the pool. This is the classic OS virtual-memory pattern applied to GPU memory.

The scheduler runs at every decode step:

Look at the active batch (currently-generating requests).
Check the waiting queue. If we have free KV blocks, admit new requests (run their prefill in this step or the next).
For each active request, run one decode step. This emits one token per request and consumes one more KV block per request as needed.
Stream tokens back to clients via SSE.
Drop finished requests from the batch (EOS token, hit max_tokens, or client disconnected). Free their blocks.
If we’re memory-pressured, preempt the longest request — either swap its KV cache to CPU memory or recompute it later. Resume when memory frees up.

The state per request is small: { request_id, prompt_tokens, generated_tokens, page_table, sampling_params, client_stream }. The KV blocks themselves live in a global pool, not per-request — that’s the whole point.

Effective batch size with continuous batching and paging is often 3–5x what static batching achieves on the same hardware. That ratio is the dollar difference between a workable platform and one that bankrupts the team.

6. Data model and storage

LLM serving is largely stateless at the request level — the heavy state is GPU memory, not durable storage. But there’s still meaningful state:

model_registry:
  model_id (PK)
  model_name
  weights_uri
  tensor_parallel_size
  context_length
  default_sampling_params

request_log:
  request_id (PK)
  tenant_id
  model_id
  prompt_tokens
  completion_tokens
  start_ts
  end_ts
  status

prefix_cache:
  prefix_hash (PK)
  kv_block_refs
  hit_count
  last_used

Storage choices:

model_registry — relational store like Postgres. Low-cardinality, low-write-rate, queried at worker startup. A KV store would work but Postgres handles the relational joins (model + tenant permissions) without ceremony.
request_log — write-heavy, append-only, queried by time range per tenant. Cassandra or ClickHouse — same shape as the clicks table in any analytics design. Drives billing and observability.
prefix_cache — in-GPU memory, with metadata in Redis. This is the system prompt optimization: when many requests share the same prefix (a long system prompt, a few-shot template), cache the KV blocks for that prefix and reuse them across requests. We come back to this in the read path.

I’d say explicitly: “The hot data lives in GPU HBM — model weights, KV cache, prefix cache. The Postgres and Cassandra stores are control plane and billing. Confusing those two is how teams end up with database calls in their inference hot path.”

7. The serving path

At 10K concurrent requests, the architecture is a layered routing system in front of a fleet of inference workers.

flowchart LR
  Client[Client] --> Gateway[API gateway]
  Gateway --> Router[Model router]
  Router --> Sched[Scheduler · model A]
  Sched --> Worker[(Worker · 2x H100)]
  Worker --> Sched
  Sched --> Client

Layers worth naming:

API gateway. Auth, rate limiting, request validation, tenant attribution. Nothing LLM-specific lives here. This is also where the SSE response is held open while tokens stream from the worker back through the stack.
Model router. Maps (tenant, model_name) to the right pool of scheduler instances. Maintains health and capacity for each pool.
Scheduler. One per model (or per model shard). Holds the request queue, runs the continuous-batching loop described in §5, and forwards decode steps to the workers. The scheduler is stateful — it owns the active batch — so it can’t be load-balanced across replicas the way a stateless service can.
Worker. Owns the GPU(s), the model weights, and the KV cache pool. Receives batched decode requests from the scheduler and returns logits.

The TTFT budget breaks down roughly as: 50ms gateway + routing, 100ms in the scheduler queue (depends on load), 300ms prefill (a 1K-token prompt on a 70B model with TP=2 is roughly that on H100), then streaming starts. The bulk of TTFT is prefill — there’s no caching path that helps unless prefixes repeat.

Prefix caching. When the system prompt is shared (a customer support template, a few-shot reasoning template), the prefill for that prefix can be cached as KV blocks and reused. The first request pays the prefill cost; the next 1000 requests with the same prefix skip prefill entirely and start decode immediately. TTFT drops from 300ms to ~50ms. This is the closest thing to a free lunch in LLM serving and it’s specific to the LLM workload — no general HTTP cache helps you here.

8. Sharding: tensor parallelism, pipeline parallelism, and request routing

Two orthogonal sharding axes, and they’re often confused.

Sharding the model across GPUs. A 70B model in fp16 weights doesn’t fit on one H100 (80GB). Two strategies:

Tensor parallelism (TP). Each layer is split across N GPUs; every forward pass requires all-reduce across them. Works best within a single node (NVLink). For 70B, TP=2 on H100 is the standard choice.
Pipeline parallelism (PP). Different layers on different GPUs; activations flow through the pipeline. Useful when crossing nodes. Adds latency but enables much larger models. For our scale, TP within a node is enough.

Sharding requests across model replicas. With 333 TP=2 shards of the 70B model, the router has to pick one. Options:

Round-robin or least-loaded — works for stateless services, but ignores prefix-cache locality.
Consistent hashing on the system-prompt prefix — sticky-routes requests with the same prefix to the same scheduler, dramatically improving prefix-cache hit rate.

I’d pick prefix-aware routing because it’s specific to this workload: prefixes repeat heavily in chat applications (everyone hits the same system prompt), and routing requests with the same prefix to the same scheduler turns a per-replica cache into something approaching a shared cache without the coherence problem.

Tip

Prefix-aware consistent hashing is one of the few design choices in LLM serving that is both non-obvious and high-leverage. A customer support application where all requests share a 2K-token system prompt can see TTFT drop from ~300ms to ~50ms with no hardware changes — just routing. This is worth calling out explicitly because it demonstrates you understand what makes LLM traffic structurally different from general HTTP traffic.

Extending the architecture:

flowchart LR
  Client[Client] --> Gateway[API gateway]
  Gateway --> Router[Model router]

  Router --> Sched1[Scheduler · 70B · shard 1]
  Router --> Sched2[Scheduler · 70B · shard 2]
  Router --> SchedN[Scheduler · 70B · shard N]
  Router --> Sched13[Scheduler · 13B]
  Router --> Sched7[Scheduler · 7B]

  Sched1 --> W1a[(GPU 0)]
  Sched1 --> W1b[(GPU 1)]
  Sched2 --> W2a[(GPU 2)]
  Sched2 --> W2b[(GPU 3)]

  classDef new fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class Sched1,Sched2,SchedN,Sched13,Sched7,W1a,W1b,W2a,W2b new

Each model has its own pool of TP-sharded scheduler+worker pairs. The router picks within a pool using prefix-aware consistent hashing.

9. Autoscaling and the cold-start problem

Autoscaling LLM workers looks like normal autoscaling but isn’t. The warmup time for a new worker is dominated by loading model weights from object storage into GPU memory — for a 70B model, this is 30 seconds to several minutes, not the seconds you’d expect from a normal Kubernetes pod spinning up.

Warning

Reactive autoscaling (trigger on high QPS or queue depth, provision, wait for the pod to be ready) fails for LLM workers because the cold-start time exceeds the duration of most traffic spikes. A burst that lasts 3 minutes cannot be absorbed by workers that take 4 minutes to warm up. This is the case where predictive scaling and standing warm-but-idle capacity are not premature optimization — they are requirements.

The implications:

Reactive autoscaling (scale up when QPS rises) is too slow. By the time a new worker is ready, the spike is over and the queue has backed up.
Predictive scaling (forecast based on time-of-day patterns) plus generous headroom is the better default.
Keep a small pool of warm-but-idle workers in reserve to absorb spikes, even if they’re underutilized most of the time.
Cache model weights on local NVMe per node, not just S3, so the bottleneck is PCIe bandwidth (~40s for 140GB) instead of network throughput (which can be 5–10x slower).

Adding the autoscaling and ingestion path:

flowchart LR
  Client[Client] --> Gateway[API gateway]
  Gateway --> Router[Model router]

  Router --> Sched1[Scheduler · 70B · shard 1]
  Router --> Sched2[Scheduler · 70B · shard 2]
  Router --> Sched13[Scheduler · 13B]
  Router --> Sched7[Scheduler · 7B]

  Sched1 --> W1[(GPU pair · TP=2)]
  Sched2 --> W2[(GPU pair · TP=2)]

  Autoscaler[Autoscaler] -.->|provisions| Sched1
  Autoscaler -.->|provisions| Sched2
  Metrics[(Metrics · Prometheus)] -.-> Autoscaler

  Sched1 -.->|usage events| Kafka[(Kafka)]
  Sched2 -.->|usage events| Kafka
  Kafka --> BillingConsumer[Billing consumer]
  BillingConsumer --> Analytics[(Analytics DB)]

  Registry[(Model registry · Postgres)] -.-> Sched1
  Registry -.-> Sched2
  ObjStore[(Model weights · S3)] -.-> W1
  ObjStore -.-> W2

  classDef new fill:#eef2f1,stroke:#2c5f5d,stroke-width:2px,color:#1f2937;
  class Autoscaler,Metrics,Kafka,BillingConsumer,Analytics,Registry,ObjStore new

Token-level usage events flow into Kafka and onward to billing — same pattern as click analytics in any other system, just with tokens_in / tokens_out instead of clicks.

10. Failure modes

I’d proactively walk through what breaks. This is where SDE III candidates distinguish themselves on this question, because the failure modes here are unusual.

A worker GPU OOMs mid-batch. With paged attention, this is rare but not impossible (a request unexpectedly generating long outputs). The scheduler preempts the longest-running request — either swap its KV cache to CPU memory and resume later, or drop it and recompute. The other requests in the batch continue. This is graceful degradation built into the scheduler, not a failure escape hatch.
Client disconnects mid-stream. Without explicit handling, the worker keeps generating and burning GPUs for a request nobody will read. The gateway has to detect the disconnect and signal the scheduler to drop the request. Without this, a malicious client can DDoS by opening and immediately closing connections.
Scheduler crashes. The scheduler is stateful — it owns the active batch and the request queue. On crash, in-flight requests fail. Mitigation: short queues, fast restart, and treating scheduler failure as a partial outage of one model shard rather than the whole platform. Other shards of the same model pick up new requests via the router.
Model weights corrupted in object storage. A worker fails to load weights and stays unhealthy. The autoscaler doesn’t replace it (the replacement will fail too). Mitigation: weight checksums verified at load time, blast-radius limited by promotion gates that validate a model on a canary worker before fleet-wide rollout.
Tenant runs away with all the capacity. One tenant submits 10K long-context requests and starves everyone else. Mitigation: per-tenant rate limits at the gateway, plus fair-share scheduling at the model scheduler so no tenant occupies more than its quota of in-flight slots.
GPU node fails. TP=2 means a single GPU failure takes down both GPUs in the shard (the model can’t run with one). The scheduler reroutes new requests to other shards; the autoscaler provisions a replacement node. In-flight requests on that shard fail and clients see 503s — these failures are not retriable transparently because the LLM output is non-deterministic.

The pattern to notice: name what fails, name what degrades gracefully, name what doesn’t. Cancellation and tenant fairness are the two failure modes most teams find out about in production.

Caution

The client-disconnect failure mode is consistently underestimated. Without explicit disconnect detection at the gateway and a cancellation signal propagated to the scheduler, a streaming-capable LLM platform is trivially DDoSable: open connections, immediately close them, repeat. The GPU keeps generating tokens for nobody. This is not a hypothetical — it is the first production incident many teams encounter. Design the cancellation path before launch, not after.

11. What I’d skip, and say I’m skipping

Time check: five minutes left. Things I’d explicitly defer:

Quantization details. Whether to serve fp16, fp8, int8, or AWQ. This is a real lever (2–4x throughput improvement is common) but it’s a model-team decision, not an architecture one. Worth a sentence, not a slide.
Speculative decoding. Another throughput lever — generate draft tokens with a small model, verify with the big one. Worth knowing exists; not load-bearing for the architecture.
Eval, A/B testing of model versions, and shadow deployments. Real systems need this. Out of scope here unless the interviewer asks.
Fine-tuning, LoRA serving, and adapter management. A separate problem shape — multi-tenant adapter serving has its own scheduler considerations.
Multi-region. Most LLM platforms are single-region or hub-and-spoke because of GPU availability, not because the design forbids multi-region.

Saying “I’d skip this, and here’s why” signals scoping discipline. The interviewer should believe you know the full surface and chose what to draw.

12. Wrap-up

One crisp sentence:

This design optimizes for GPU utilization through continuous batching and paged KV cache, accepting per-shard stateful schedulers as the price for the throughput; the architecture’s hot path is GPU memory management, not request routing.

That framing — naming GPU memory management as the architectural center of gravity — is the kind of articulation that lands.

What separates SDE II from SDE III on this question

SDE II usually names a model server (vLLM, TGI), draws a load balancer in front, mentions GPUs, and gestures at autoscaling.
SDE III identifies KV cache memory as the binding constraint in the capacity estimate, walks through continuous batching and paged attention at re-implementation depth, calls out cancellation and prefix-cache routing as workload-specific concerns, names cold-start asymmetry as a reason reactive autoscaling doesn’t work, and proactively scopes out quantization and speculative decoding.
Staff/Principal frames the platform as a portfolio of trade-offs across cost, latency, and reliability axes — and makes those trade-offs explicit rather than converging on a single “right” design. Proactively identifies the inflection points: at what request volume does prefix caching become worth the routing complexity? At what model size does pipeline parallelism replace tensor parallelism? Asks “how does on-call debug a TTFT regression at 3am?” — which surfaces the need for per-request tracing, scheduler queue depth metrics, and KV cache utilization dashboards as design requirements, not afterthoughts. Surfaces the missing requirement of GPU procurement lead time and what that means for capacity planning. Reasons about build vs buy at the inference framework layer (vLLM vs TRT-LLM vs a managed API) in terms of team ownership, hardware dependency, and long-term maintenance cost — not just benchmarks. Asks “when does this break?” for both the scheduler (what happens if the continuous batching loop falls behind?) and the business (what does a 2x GPU cost increase do to the unit economics?).

The difference isn’t knowledge of the buzzwords. It’s naming the binding constraint and showing why every other decision falls out of it.