What it is
In a distributed system, dependencies are slow or down far more often than the code that calls them assumes, and a slow dependency is more dangerous than a dead one: it ties up threads and connections until the caller fails too. Resilience patterns contain failure so it does not cascade: timeouts bound the wait, retries (with backoff and jitter) ride out transient errors, circuit breakers stop calling something that is failing, bulkheads isolate resources, and load shedding and graceful degradation keep the system useful under overload.
Important
Key takeaway: Every call needs a deadline. Retries need backoff, jitter, and a budget, and only work on idempotent operations. Isolate dependencies with bulkheads and breakers so one slow service cannot take down the rest, and decide in advance what the system does when a dependency is gone: degrade, do not fail.
When you care
“What happens when this downstream service is slow or down?” is the follow-up to almost every microservice design, and the strong answer names more than “retry”.
Note
Interview signal: Saying that retries amplify load, with the arithmetic (three layers each retrying three times is 27 calls at the bottom), marks experience. Adding jitter, a retry budget, and “only retry idempotent calls” completes the answer.
Timeouts and deadlines
Without a timeout, a call to a hung dependency waits forever, holding a thread. A timeout per call bounds it. Better, a deadline set at the edge is propagated through the call chain, and each hop spends from the same budget, so the last hop does not keep working on a request the first hop has already abandoned. Set timeouts from measured latency (a bit above the p99), not from a guess.
Worked example: how a slow dependency exhausts a pool. A service has 100 worker threads and its dependency usually answers in 100 ms, so its capacity is 100 ÷ 0.1 = 1,000 requests per second (Little’s law). The dependency slows to 2 seconds: capacity falls to 100 ÷ 2 = 50 requests per second. At the old 1,000 requests a second arriving, the pool is full in a fraction of a second, every request waits, and the service looks dead although it is healthy. A 200 ms timeout would have kept capacity at 500 requests a second and failed fast.
Retries
Retry transient failures (timeouts, 503s, connection resets) and never permanent ones (400s, authorization failures).
- Exponential backoff: wait longer after each failure (for example 100 ms, 200 ms, 400 ms, capped).
- Jitter: randomize the wait (
sleep = random(0, min(cap, base × 2^attempt))), so many clients that failed together do not all retry at the same instant (a thundering herd that re-crashes a recovering service). - A retry budget: limit retries to a fraction (say 10%) of normal traffic, or a maximum per request, so retries cannot multiply load.
- Idempotency: a retried non-idempotent call (a payment) can execute twice, so retry only idempotent operations or use an idempotency key.
- Retry in one place. If every layer retries, the amplification is multiplicative.
Worked example: retry amplification. A request passes through three layers, and each retries a failing call up to 3 times in total (one try plus two retries). If the bottom service is down, the layer above makes 3 calls, each of those layers makes 3 more: 3 × 3 × 3 = 27 calls hit the failing service for one user request. A service already struggling gets 27× the load it did, which keeps it down. This is how a brief blip becomes a retry storm.
Circuit breakers
A circuit breaker wraps calls to a dependency and tracks failures. It has three states:
| State | Behavior |
|---|---|
| Closed | Calls pass through; failures are counted |
| Open | Calls fail immediately (or return a fallback) without touching the dependency, for a cool-off period |
| Half-open | After the cool-off, a few trial calls go through; success closes the breaker, failure reopens it |
A typical rule: open when the error rate exceeds 50% over a 10-second window with at least 20 requests, stay open for 30 seconds, then probe. An open breaker gives the dependency room to recover and gives callers an instant error or fallback instead of a slow timeout.
Bulkheads
A bulkhead isolates resources so a failure in one place cannot drain a shared pool, as ship compartments contain flooding. Give each dependency its own thread pool, connection pool, or concurrency limit. If the recommendations service hangs, it consumes only its own pool of 20 threads, and checkout, which has its own pool, keeps working. Without bulkheads one slow dependency can exhaust a shared pool and stop everything.
Load shedding, backpressure, and degradation
- Load shedding: when overloaded, reject excess requests quickly (a 503 or 429) rather than queueing them until everything times out. Prioritize: shed low-value or non-critical requests first and protect critical paths.
- Backpressure: push the signal upstream (a bounded queue that rejects when full, a client that slows down) so overload does not accumulate silently in memory.
- Graceful degradation: define a lower-quality response for each dependency failure, such as serving cached or stale data, a default recommendation list, or a read-only mode, instead of an error.
- Timeouts at the edge, rate limits, and admission control keep overload out of the deep stack.
Warning
Production reality: Metastable failures. Some systems stay broken after the trigger is gone: a brief overload causes timeouts, the timeouts cause retries, the retries sustain the overload, and the system never recovers without human intervention (shed load, disable retries, restart). Retry budgets, breakers, and load shedding are what stop a short incident from becoming a long one.
When to pick what
| Problem | Pattern | Why |
|---|---|---|
| A call might hang | A timeout, and a propagated deadline | Bounds waiting and thread use |
| Transient errors | Retries with exponential backoff, jitter, and a budget | Rides out blips without a storm |
| A dependency is failing | A circuit breaker with a fallback | Fail fast and give it room to recover |
| One slow dependency starves everything | Bulkheads (separate pools) | Contains the damage |
| Overload | Load shedding and backpressure | Protects the system and critical paths |
| A dependency is gone | Graceful degradation (cache, defaults, read-only) | Useful beats correct-or-error |
| A slow replica inflates p99 | Hedged requests | Send the read to an extra replica when the first responses are late |
Caution
Common mistake: Adding retries without timeouts, backoff, jitter, or idempotency. Retries without a timeout wait forever; without backoff they hammer a struggling service; without jitter they synchronize; without idempotency they duplicate side effects. Retries are a load multiplier, so budget them.
Related
- Walkthrough: Designing a Flash Deal System — load shedding and degradation under a traffic spike, with the correctness core protected.
- Walkthrough: Designing a Notification System — retries with backoff against third-party providers that throttle and fail.
- Reference: Idempotency and the Outbox Pattern — the prerequisite for any retry: making the repeated call harmless.