What it is
A distributed lock gives one client at a time the right to do something (run the nightly job, write a file, act as leader) across machines that share no memory. The hard part is not granting the lock; it is deciding what breaks when the system disagrees about who holds it. Locks come in two kinds: an efficiency lock (avoid duplicate work, where an occasional double grant is harmless) and a correctness lock (a double grant corrupts data). The design differs sharply.
Important
Key takeaway: Safety is layered. Consensus gives agreement on who holds the lock; fencing tokens stop a stale holder that does not know it lost the lock. A lock service alone cannot make a paused process stop acting, so a correctness lock without fencing is not safe.
When you care
Any design with a singleton job, leader election, or exclusive access to a shared resource raises “what if the lock holder crashes, or pauses?” The trap is answering with a Redis lock and a TTL and stopping there.
Note
Interview signal: Saying “a TTL is a clock opinion; the resource’s token check is a fact” shows you understand why timeouts cannot fix a stale holder. Naming the garbage-collection pause as the concrete failure is the detail that lands.
Three ways to build one
| Dimension | Single Redis | Redlock (majority of 5 Redis) | Consensus sessions (ZooKeeper, etcd, Chubby) |
|---|---|---|---|
| Double grant on node failover | Yes: asynchronous replication can lose the lock | No (majority) | No (replicated state machine) |
| Depends on clock behavior | TTL only | Yes, the safety argument depends on bounded drift and no pauses | Heartbeat timing affects liveness, not safety |
| Stale holder after a pause | Unprotected | Unprotected | Unprotected without fencing |
| Acquire latency | About 1 ms | 5 parallel round trips | A consensus commit, a few milliseconds |
| Right for | Efficiency locks | Few honest niches | Correctness locks and leader election |
Single Redis. SET lock NX PX ttl with a unique value, released with a compare-and-delete script. Sound on one healthy node. The
hole: replication is asynchronous, so the primary can acknowledge the lock and die before the replica sees it; the replica is
promoted and the next client acquires the “same” lock. Two holders, each with a legitimate grant.
Redlock. Acquire on a majority of five independent Redis nodes and count elapsed time against the TTL. It removes the failover hole but relies on clocks drifting within bounds and clients not pausing between the check and the use, which is exactly what critics attack.
Consensus-backed sessions. Lock state lives in a replicated state machine (Raft or ZAB). Clients hold sessions kept alive by heartbeats; a lock is an ephemeral record owned by a session, and session expiry deletes it atomically, in one agreed order. This refuses rather than double-grants during a partition, which is the right bias for correctness. But note the caveat: it still does not stop a stale holder.
Acquire path, sessions, and wait queues
- A client opens one session per process (not per lock) and sends
acquire("jobs/reindex"). - The leader commits the command through consensus. The state machine applies it deterministically: if the lock is free, assign it to the session and increment the lock’s fencing counter.
- The grant returns
{token: 34}. If the lock was held, the request joins an ordered wait queue and the client sets a watch; there is no polling. - The session heartbeats (for example every 3 seconds). If it misses the session timeout (about 10 seconds), the state machine expires the session and releases all its locks, notifying the head of each wait queue.
Why a wait queue with ordered grants: on release, exactly one waiter is notified. The alternative, notifying everyone and letting them race to re-acquire, is a thundering herd that produces grant storms at scale.
Caution
Common mistake: Tying liveness to a per-lock TTL that the holder must keep refreshing. Every lock becomes its own timer, refresh failures race with the work, and a client holding 50 locks runs 50 renewal loops. Sessions invert it: one heartbeat proves the client alive, and the state machine owns every consequence of its death.
Fencing: the other half of correctness
The resource, not the lock service, enforces exclusion by rejecting a lower token.
The failure on a timeline.
| Time | Event |
|---|---|
| t0 | A acquires the lock, token 34, and starts work |
| t1 | A hits a 25-second stop-the-world garbage-collection pause |
| t2 | A misses heartbeats; the service expires its session and releases the lock |
| t3 | B acquires the lock, token 35, and starts work |
| t4 | A’s pause ends. A believes it still holds the lock and writes to the resource |
Without fencing, t4 corrupts the resource, and every component behaved correctly: the service expired a dead-looking session and A was merely slow. No timeout tuning fixes this; a longer timeout trades t2’s promptness for a wider t4 window.
With fencing, every write to the protected resource carries the grant’s token. The resource remembers the highest token it has seen (B’s first write recorded 35) and rejects anything lower. A’s write arrives with 34 and is refused; A learns it lost the lock the only way a paused process can, by being told after the fact. The mechanism is one integer comparison, but it relocates the guarantee: mutual exclusion stops being a promise about the future (“nobody else holds this”, unknowable under pauses) and becomes a check in the present (“this token is stale”, a fact).
Warning
Production reality:
Fencing requires the resource to cooperate: a conditional write that checks token order, which is trivial in a database
(WHERE token <= 34) and supported by some object stores through preconditions. Many real resources cannot check anything (a
third-party API, an SMTP server). For those, no distributed lock is safe, and the honest mitigations are idempotent operations
and reconciliation after the fact.
Leader election with the same mechanism
Election is the acquire path with a longer lease:
- Every candidate tries
acquire("election/orders-primary"); the winner is leader and the losers watch. - The fencing token doubles as the epoch number. Every action the leader takes downstream carries it.
- Leader death means session expiry, then the next waiter is promoted with epoch + 1. Downstream systems reject epoch n once they have seen n + 1, so a brief overlap of two self-believed leaders (split brain) is tolerable: the old leader’s actions bounce.
One detail interviewers probe: a leader that loses contact with the lock service must stop acting before its session would
expire, stepping down at session timeout - safety margin by its own clock. That margin is the one place a clock assumption
legitimately remains, and it errs toward unavailability (a gap with no leader) rather than unsafety (two leaders).
When to pick what
| Situation | Choose | Why |
|---|---|---|
| Avoid duplicate work; a rare double run is harmless | A single Redis lock with a TTL | Simple and fast |
| Correctness matters (money, shared state) | Consensus sessions plus fencing tokens | Agreement plus a stale-holder defense |
| Leader election for a service | Consensus lock; the token is the epoch | Same mechanism, longer lease |
| The resource cannot check a token | Redesign: idempotent operations, or avoid the lock | Nothing makes this safe |
| Many waiters on one lock | An ordered wait queue with watches | No thundering herd |
Related
- Walkthrough: Designing a Distributed Lock — the full lock-service design, sessions and wait queues included.
- Walkthrough: Designing a Distributed Job Scheduler — at most one active execution per job, with leases and fencing.
- Reference: Consensus and Leader Election — the agreement layer underneath a correctness lock.