What it is
A load balancer spreads requests across a pool of servers so no one is overloaded and a dead server is skipped. Service discovery answers the question that comes first: which servers exist right now, and where are they? In a system where instances come and go every few minutes, the two together are how a client reaches “the orders service” without knowing any address.
Important
Key takeaway: Pick the layer (L4 for raw throughput and non-HTTP, L7 for routing by content), the algorithm by how uneven your requests are (least-loaded or power-of-two-choices beats round robin when requests vary), and always pair it with health checks and connection draining. Keep headroom so a lost instance does not overload the rest.
When you care
Every design has a load balancer box, and the follow-ups are “what happens when a server dies”, “how do you keep a user on one server”, and “how do services find each other”. The classic trap is long-lived HTTP/2 connections (gRPC) behind a layer-4 balancer.
Note
Interview signal: Naming the gRPC trap (a layer-4 balancer routes on connection, but gRPC multiplexes many requests over one long-lived connection, so everything lands on one backend) shows real experience. The fix is a layer-7 proxy or client-side balancing.
Layer 4 versus layer 7
| Layer 4 (transport) | Layer 7 (application) | |
|---|---|---|
| Decides by | IP address and port | The request: host, path, headers, cookies |
| Speed | Very fast; little per-request work | Slower: parses the request |
| TLS | Can pass through | Usually terminates TLS |
| Routing | Per connection | Per request: path-based, header-based, canary splits |
| Good for | Throughput, non-HTTP protocols, fixed addresses | HTTP and gRPC APIs, microservice routing, WebSockets |
| Examples | A network load balancer, HAProxy in TCP mode | An application load balancer, Envoy, NGINX |
The gRPC trap. gRPC runs over HTTP/2, which multiplexes many calls over one long-lived TCP connection. A layer-4 balancer picks a backend when the connection opens, so all that client’s traffic goes to one server, and adding servers does not help. Use a layer-7 proxy that balances each request, or client-side balancing where the client holds connections to all backends.
Balancing algorithms
| Algorithm | How | Use when | Weakness |
|---|---|---|---|
| Round robin | Each backend in turn | Uniform, short requests | Ignores load and request cost |
| Weighted round robin | Proportional to a weight | Backends of different sizes | Weights are static |
| Least connections | The backend with the fewest open requests | Requests vary in duration | Needs shared state; stale under many balancers |
| Least response time (latency-aware, EWMA) | Favors the fastest recent backend | Heterogeneous or noisy backends | Can pile onto a fast-but-failing node |
| Power of two choices | Pick two at random, send to the less loaded | Many balancers, no shared state | Slightly less optimal than a global view |
| Consistent hashing | Hash a key (user, session, cache key) to a backend | Cache locality or stickiness | Uneven load for hot keys |
Worked example: why two choices is enough. With plain random assignment of n requests to n servers, the busiest server ends up with about log n ÷ log log n requests (several times the average at large n). Picking two random servers and sending to the less loaded drops the maximum to about log log n, an exponential improvement for one extra probe, and it needs no shared global state, which is why it suits fleets of independent balancers.
Stickiness (the same user to the same backend) is needed only when state lives on the server: use a cookie or consistent hashing on a key. Prefer stateless servers with state in a shared store; stickiness fights even load and complicates draining.
Health checks, draining, and headroom
- Active health checks probe each backend (an HTTP path, a TCP connect) on an interval and remove it after N failures; add it back after M successes. Passive checks watch real traffic and eject a backend on errors or timeouts. Use both.
- A health endpoint should check what makes the instance able to serve (its critical dependencies), but beware a shallow check that passes while the app is broken, and a deep check that fails every instance when one shared dependency blips (a cascading outage).
- Connection draining: when removing a backend (a deploy, a scale-in), stop sending new requests but let in-flight ones finish within a time limit.
- Slow start: ramp a new backend’s traffic gradually, so a cold cache or an unwarmed runtime is not hit with full load.
Worked example: headroom. Six instances at 70% utilization. If one dies, the other five carry its load: 0.70 × 6 ÷ 5 = 84%, and at 84% queueing delay is already steep. If the fleet had run at 85% utilization, losing one would push the rest to 85% × 6 ÷ 5 = 102%, over capacity, so a single failure becomes an outage. Size for N + 1 (survive one loss) or N + 2 across zones, not for average load.
Global load balancing
Across regions, traffic is steered by DNS (geographic or latency-based answers, with health checks that remove a failed region) and anycast (the same IP announced from many places, so the network routes to the nearest). DNS answers are cached for their time-to-live, so failover is as slow as the TTL; short TTLs for failover records trade DNS load for agility.
Service discovery
| Approach | How | Strengths | Weaknesses |
|---|---|---|---|
| DNS | Names resolve to the current instances (a headless service or a record per instance) | Universal, no client library | Caching delays updates; clients may pin one address |
| Server-side (a load balancer in front) | Clients call one stable address; the balancer knows the pool | Simple clients | An extra hop; the balancer is critical infrastructure |
| Client-side | Clients query a registry (Consul, etcd, Eureka) and balance themselves | No extra hop; smart clients (latency-aware, zone-aware) | A client library in every language; registry dependency |
| Service mesh (sidecar proxies) | A proxy next to each service handles discovery, balancing, retries, mutual TLS | Uniform behavior without app changes | Operational complexity and latency |
Instances register themselves (or are registered by the platform) with a health status and a heartbeat; stale entries expire. The registry itself is a small, critical, consensus-backed store. Cache its answers in the client so a registry blip does not stop traffic, and route around instances that fail locally, not only ones the registry has removed.
Warning
Production reality: The balancer and the registry are single points of failure unless you make them redundant. Run balancers in several zones, make clients tolerate a stale registry, and test what happens when the registry is unavailable. A frequent real outage is a health check that marks every instance unhealthy at once, after which a “fail open” policy (serve to all) is better than serving to none.
When to pick what
| Situation | Choose | Why |
|---|---|---|
| HTTP APIs, path routing, canary releases | A layer-7 balancer | Per-request decisions |
| Maximum throughput, non-HTTP, fixed IPs | A layer-4 balancer | Fast, simple |
| gRPC or other HTTP/2 multiplexing | Layer 7 or client-side balancing | Avoid the one-connection trap |
| Requests of very different cost | Least connections or power of two choices | Round robin ignores cost |
| A cache tier or session affinity | Consistent hashing | Same key, same node |
| Services find each other | A registry with client-side balancing, or a mesh | Instances change constantly |
| Survive a lost instance | Size for N + 1 headroom | Utilization after failure is what matters |
Caution
Common mistake: Sizing for average load. The fleet’s capacity after losing an instance (or a zone) is what decides whether a failure is an incident. Also: a health check that depends on a shared downstream can eject every instance together, turning a partial outage into a total one.
Related
- Walkthrough: Designing a Chat System — long-lived connections behind gateways, plus the registry that routes a message to the right socket.
- Walkthrough: Designing a Distributed Cache — consistent hashing as the balancing algorithm when each key must land on one node.
- Reference: Resilience Patterns — what callers do once the balancer has routed them to a slow or failing backend.