Technologies referenced in this case study: ZooKeeper & etcd · Apache Kafka · DynamoDB · PostgreSQL · Redis

Related: Distributed Coordination · Service Discovery · Replicated Data Store · Distributed Job Scheduler · Consistency Models · Build vs Buy

How to Use This Case Study#

Organized for interview use first, reference second. The algorithm is the smallest part of this interview; the fault lines and the "should we run this at all" judgment are the largest.

Without consensus-backed election + fencing:
t=0:       Scheduler A is leader (it holds a Redis key with a 10s TTL).
t=+2s:     A enters a 15s stop-the-world GC pause.
t=+12s:    Key expires. Scheduler B acquires it and becomes leader.
t=+17s:    A resumes. Its local state says "I'm leader". It never re-checked.
t=+17s:    A and B both dispatch the nightly billing job.
t=+18s:    14,000 customers invoiced twice. Finance on the phone by 9am.

With consensus-backed lease + fencing token:
t=0:       A holds leadership with fencing token 41 (a monotonically increasing epoch).
t=+2s:     A pauses for 15s.
t=+12s:    Lease expires. B elected with token 42. B writes to the job store with token 42.
t=+17s:    A resumes and tries to dispatch with token 41.
t=+17s:    Job store rejects: "token 41 < highest seen 42". A steps down.
t=+17s:    Zero duplicate invoices. One log line and a metric increment.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What This Interview Actually Tests#

Consensus is not a Raft-recitation question. Anyone can draw leader → followers → majority ack.

It is a correctness-boundary and blast-radius question that tests:

• Whether you know what a consensus service actually guarantees — and the gap between "the service knows who the leader is" and "the old leader has stopped acting"
• Whether you size quorums against the failures you actually expect (a zone, a region, a bad deploy)
• Whether you keep consensus off the hot path and keep its data small
• Whether you recognize that this becomes the company's most shared dependency, and design its blast radius accordingly

The key insight: Consensus gives you agreement inside the cluster. It does not give you mutual exclusion outside it. Every lock and leader election leaks at the boundary where a paused or partitioned process still believes it's in charge — and only fencing at the resource closes that gap.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll design the coordination service — the Chubby/etcd shape — used for leader election, locks and config by other systems. It holds megabytes, not terabytes, and handles hundreds of writes per second, not millions. If you want consensus as the replication layer of a database, that's multi-Raft sharding, and I'd treat it as a different design."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: These three systems define the vocabulary interviewers use. Cite them by mechanism, not by name only.

Google Chubby — Coarse-Grained Locks as a Service#

Google's Chubby paper (2006) describes a lock service built on Paxos, typically five replicas per cell, used by systems like GFS and Bigtable for master election and for storing small amounts of metadata. Chubby deliberately offers coarse-grained locks held for hours or days, not fine-grained per-request locks. Clients hold sessions maintained by KeepAlives, cache data with server-driven invalidation, and receive a sequencer with a lock that downstream servers can check — Chubby's form of a fencing token.

Staff insight: The paper's most-quoted lesson is that engineers used Chubby as a name service and small database far more than as a lock service, and that load was dominated by reads and KeepAlives. Design your coordination service for how it'll be abused, not how it's specified.

etcd in Kubernetes — The Cluster's Single Source of Truth#

Every Kubernetes object — pods, services, deployments, secrets — lives in etcd, a Raft-based key-value store. Controllers don't poll; they watch resources from a revision and reconcile. The API server fronts etcd so that thousands of clients don't hit it directly. etcd documents a default storage quota of 2 GiB (configurable, with ~8 GiB suggested as a practical maximum) and a request size limit of about 1.5 MiB; when the quota is exceeded, etcd raises a NOSPACE alarm and rejects writes until compaction and defragmentation free space.

Staff insight: Kubernetes shows the right architecture — a caching, watch-multiplexing layer (the API server) in front of consensus — and the right failure: when etcd is slow, the whole control plane is slow, but running pods keep serving. Control plane down ≠ data plane down.

Apache Kafka — Removing ZooKeeper (KRaft)#

Kafka historically stored cluster metadata (brokers, topics, partition leaders) in ZooKeeper. KIP-500 replaced it with KRaft, a Raft-based metadata quorum inside Kafka itself, and Kafka 4.0 removed ZooKeeper mode. Motivations publicly cited include operating one system instead of two, faster controller failover, and supporting far more partitions because metadata is a replicated log that brokers consume rather than state loaded from ZooKeeper on failover.

Staff insight: "Build vs buy" for consensus flipped only when coordination became core to the product's scaling limit, with a mature team and years of runway. That's the bar for building: consensus is your product, not your dependency.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: Three separations matter. The access layer coalesces watches and enforces quotas so 10,000 clients look like 50 to the core. The consensus core is small and zone-spread so any single zone loss leaves a majority. The protected resource enforces fencing tokens — that's where mutual exclusion is actually guaranteed. The metrics panel watches the things that cause real incidents: fsync latency, leader churn, DB size vs quota and watcher count.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 20 minutes explaining Raft terms, votes and log matching — accurately — and never reach the question the interviewer is waiting for: "The old leader was paused. What stops it from doing damage?" Compress the protocol to 3 minutes and spend the rest on fencing, placement, reads and blast radius.

Phase 1: Requirements & Framing (2–3 minutes)#

State the scope in one breath:

"We need a service that lets other systems elect leaders, take locks, register membership and distribute configuration — with a guarantee that they agree even through node failures and partitions."

Then clarify the intent and cap it:

"Is this a coordination service for other systems, or the replication layer inside a database? I'll assume coordination: small data — under a few GB total — writes in the hundreds to low thousands per second, reads and watches from up to tens of thousands of clients. Correctness is non-negotiable for leader election and locks; config can be slightly stale at clients."

Then name the failure envelope:

"What must we survive — a node, a zone, or a region? I'll assume a zone, within one region, and treat region survival as a separate decision because it changes write latency by 30–50×."

🎯 Staff Move: Capping data size and write rate in the first two minutes prevents the most common drift in this interview — someone deciding to store user data in the coordination service by minute 25.

Phase 2: Core Entities & API (1–2 minutes)#

• Key: path-like name, value (≤ 1 MiB), create_revision, mod_revision, version, optional lease_id
• Lease / Session: lease_id, TTL (e.g., 10s), owner; keys attached to a lease disappear when it expires
• Revision: a cluster-wide monotonic counter incremented on every write — the basis for fencing, watches and consistent snapshots
• Watch: subscription from a revision on a key or prefix

Put(key, value, lease?, if_mod_revision?)   → { revision }          # CAS via if_mod_revision
Get(key | prefix, consistency=linearizable|serializable, at_revision?) → { kvs, revision }
Delete(key, if_mod_revision?)
Txn(compare[], then_ops[], else_ops[])       → { succeeded, revision }
LeaseGrant(ttl) → lease_id ;  LeaseKeepAlive(lease_id) (stream) ;  LeaseRevoke(lease_id)
Watch(key | prefix, from_revision)           → stream of events
# Recipes on top:
Campaign(election, lease) → { leader_key, fencing_token = create_revision }
Lock(name, lease)         → { fencing_token }

🎯 Staff Move: "The API is deliberately tiny — CAS, leases and watches. Elections and locks are client-side recipes on these primitives. The fencing token falls out for free: it's the revision at which the leader key was created, and revisions never go backwards."

Phase 3: High-Level Architecture (≤5 minutes)#

Walk the write path in 90 seconds:

1. Client sends Put to the proxy; proxy forwards to the current leader.
2. Leader appends to its log, fsyncs, sends AppendEntries to 4 followers in parallel.
3. Once 2 followers (3 of 5 including the leader) have fsynced, the entry is committed.
4. Leader applies to its state machine, bumps the revision, responds, and notifies watchers.
5. Latency ≈ one intra-region RTT + one fsync ≈ 2–5ms zone-spread.

Key points to hit:

1. Five voters, three zones, 2-2-1 — survives a zone
2. Proxy layer — clients never connect directly to consensus nodes at 10K scale
3. Leases for liveness, fencing tokens for safety
4. Off the hot path — clients cache; a coordination outage degrades control, not traffic
5. Buy it — etcd or ZooKeeper; the design work is placement, quotas, clients and operations

🎯 Staff Move: After drawing: "This is the part that's well-understood. The incidents happen elsewhere — at the resource that didn't check the token, the disk that got slow, the 10,000 watchers reconnecting. That's where I'd like to go deep."

Phase 4: Transition to Depth (1 minute)#

"Three areas decide whether this is safe in production: fencing — what happens when a leader doesn't know it's been replaced; quorum placement and gray failures — slow disks and flapping leaders; and read semantics and blast radius — who can depend on this and how. Which is most interesting to you?"

Default if no preference: fencing. It's where most candidates are weakest and most real-world bugs live.

Phase 5: Deep Dives (25–30 minutes)#

Deep dive 1: Leases and fencing (7–8 min)

"A lease says 'you're leader until T' — by the server's clock. The holder checks its own clock, and it can be wrong in two ways: clock drift, which is bounded if we use monotonic clocks and a safety margin, and process pauses, which aren't bounded at all — GC, VM live migration, page faults. So I make leadership safe at the resource: each leader gets a fencing token equal to the revision of its leader key. Every write to the job store carries the token; the store keeps the max token it has seen per resource and rejects lower ones. A paused ex-leader wakes up and gets rejected on its first write."

Quantify: "Lease TTL 10s, renewal every 3s, and the holder stops acting at 10s minus a safety margin of 1–2s measured on its own monotonic clock. That handles drift. Fencing handles the pause."

Who pays: "Every team that owns a protected resource has to implement the token check. That's the cost — and I'd pay it with a shared library, not by skipping it."

Deep dive 2: Quorum placement and gray failure (6–7 min)

"2-2-1 across three zones. Losing any zone leaves ≥ 3 voters. Losing a zone plus one more node elsewhere still leaves a majority only if the lost zone was the 1 — so we survive 'one zone' reliably and 'zone plus node' partially. The gray failure I worry about more is a leader with a slow disk: fsync p99 goes from 2ms to 400ms, heartbeats stall, followers time out, elect a new leader, and the old one — no longer overloaded — rejoins. If the slow disk is on a node that keeps winning elections, we flap every few seconds. etcd mitigates with pre-vote and check-quorum; I'd add an alert on leader_changes_total > 3 in 15m and a runbook to remove the sick node."

Deep dive 3: Reads and watches (5–6 min)

"Three read modes. Linearizable: ReadIndex — the leader confirms it's still leader with a heartbeat round, then serves once applied up to that index; costs one RTT, no disk write. Serializable: any member serves from local state — may be stale, fine for dashboards. Watches: clients subscribe from a revision and receive every change in order. For config at 10K clients, watches via the proxy; for leader discovery, watches; for 'am I still leader before I act', I don't read — I fence."

Deep dive 4: Blast radius and ownership (4–5 min)

"This cluster becomes the thing everything depends on. So: per-tenant key prefixes with quotas on keys, bytes and write rate; clients that cache the last-known config and keep serving if the cluster is unavailable; separate clusters per criticality tier — the Kubernetes control plane doesn't share etcd with feature flags. The platform team owns the pager; tenants own their usage, and a tenant exceeding quota gets throttled, not the cluster."

Phase 6: Wrap-Up (2–3 minutes)#

"The protocol guarantees agreement inside the cluster. Safety outside it comes from fencing at the resource. Availability comes from placement against real failure domains, and from keeping the cluster off the request path. And the biggest risk isn't consensus at all — it's that everything ends up depending on one cluster."

The build-vs-buy closer:

"I'd run etcd, not write Raft. If our product were coordination — like Kafka's move to KRaft — we'd staff a team for years and use formal methods and deterministic simulation testing. For everyone else, the work is operations, quotas and client libraries."

🎯 Staff Move: End with "who else depends on this, and what happens to them when it's down." That's the question the interviewer wants to hear you ask.

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Consensus is the interview where textbook knowledge is abundant and production judgment is rare. Most candidates can describe Raft; few have debugged a leader that flaps because one disk is slow, or a lock that "worked" in every test and double-processed during a 20-second GC pause. The Staff-level signal is recognizing that the protocol is a solved component and that the boundaries — clocks, pauses, placement, clients, quotas, and the organization's growing dependence on it — are the design.

It also tests restraint. The strongest answers often reduce the scope: "We don't need a lock here; a conditional write on the row is enough." Coordination is expensive — every write costs a majority fsync — and it becomes a shared single point of failure. Knowing when not to use it is part of the answer.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"Your leader process freezes for 30 seconds — longer than any timeout you've set — and then resumes mid-operation. Walk me through exactly what prevents it from corrupting state."

A candidate who answers "the lease expired so it's not leader" has described the coordination service's view. A candidate who answers "its next write carries token 41, the store has seen 42, the write is rejected, and the process exits on the rejection" has described the system's view. Only the second is safe.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Coordination primitives → safety at the edges

• Constraint: at most one effective actor per role at any time
• Strategy: leases for liveness, fencing tokens for safety, coarse-grained locks held for seconds to hours
• Failure mode: split-brain at the resource when fencing is absent
• Who pays for imperfection: the team whose data got written twice

Configuration / metadata store → fan-out and freshness

• Constraint: tens of thousands of readers; changes must propagate in seconds and in order
• Strategy: watches from a revision via a proxy layer; clients cache and survive outages with last-known-good
• Failure mode: watch storms on reconnect; bad config propagating to everyone in seconds
• Who pays: every service that consumes the config — a bad flag is a global incident

Replicated state machine for data → throughput and sharding

• Constraint: millions of keys, high write throughput, horizontal scale
• Strategy: multi-Raft — thousands of small consensus groups (one per range), a placement driver, range splits and merges (CockroachDB, TiKV, and Spanner's Paxos groups follow this shape)
• Failure mode: hot ranges, rebalancing storms, cross-range transactions
• Who pays: the database team; this is a database design interview, not a coordination one

2.2 When NOT to Use a Consensus Service#

• Idempotent work where duplicates are merely wasteful. Two workers both regenerating a cache is fine. Use a best-effort lock in Redis or none at all.
• Mutual exclusion on a single row in one database. A conditional write (UPDATE … WHERE version = ?) or SELECT … FOR UPDATE in PostgreSQL, or a condition expression in DynamoDB, already gives you linearizable CAS and the database is the resource — no fence gap. Adding ZooKeeper here adds a failure domain for nothing.
• Leader election for a single-writer queue consumer. Kafka's consumer group protocol already assigns partitions exclusively; don't layer another election on top.
• High-volume data. Job payloads, user sessions, metrics — anything beyond small metadata. The coordination store keeps everything in memory and in every snapshot.
• Rate limiting or counters on the request path. Every increment is a majority fsync; use local-first counters (Rate Limiting).
• Cross-region config where seconds of staleness are fine. Replicate asynchronously from a regional source of truth rather than stretching a quorum across regions.

🎯 Staff Insight: "The best coordination is the coordination you don't need. If the resource itself can do a compare-and-swap, the lock and the fence collapse into one operation — and that's strictly safer than any external lock."

2.3 What the Interviewer Leaves Underspecified#

• Which failure domain must be survived — node, zone, region
• Who the clients are — 50 controllers or 50,000 sidecars changes the access layer entirely
• Whether locks guard correctness or efficiency — decides whether fencing is mandatory
• Data size and write rate — kilobytes of leader keys vs gigabytes of "metadata" someone plans to add
• Read consistency needs — is stale config for 2 seconds acceptable?
• Multi-tenancy — is this one team's cluster or the company's?

2.4 Precise Terminology#

🎯 Staff Insight: When an interviewer says "strongly consistent", ask: "Linearizable reads, or linearizable writes with possibly stale reads? etcd and ZooKeeper differ by default — ZooKeeper reads can be stale unless you sync first."

3. The Five Fault Lines#

Each fault line has a protocol-level answer everyone knows and an operational/organizational answer that decides the level.

3.1 Fault Line 1: Safety vs Liveness — Quorum Sizing & Placement#

The tension: More voters in more failure domains survive more failures but make every write wait on a slower majority and make elections slower and more fragile. Fewer, closer voters are fast but die with a zone or region.

Staff default: "Five voters, three zones, 2-2-1, one region. Region survival for coordination is usually the wrong goal — most consumers are region-scoped anyway. I'd rather have one cluster per region and replicate the few global keys asynchronously."

When to deviate:

• Global control planes (global DNS control, identity, cross-region leader for a globally consistent database): region-spread quorum, accept the latency, keep write rates in the tens per second.
• Tiny, non-critical clusters (a team's internal scheduler): 3 nodes; the extra two voters aren't worth the operations.
• Read-heavy scale: add learners/observers (non-voting) rather than voters — reads scale, writes don't slow.

🧭 Principal Move: "Quorum placement is an org policy, not a per-cluster choice. I'd publish three tiers — zone-survivable (default), region-survivable (by exception, with a latency sign-off), and single-zone (dev only) — and have the platform provision only those shapes."

❌ Common L5 Trap: "Five nodes, so we survive two failures." Then five nodes placed 3+2 across two zones: lose the 3-node zone and quorum is gone. Node counts without failure-domain placement are arithmetic, not availability.

3.2 Fault Line 2: Leases vs Fencing#

The tension: Leases let a leader act without a round trip per operation, which is fast — but correctness depends on assumptions about clocks and pauses that production violates. Fencing tokens are safe regardless of timing, but every protected resource must implement a check.

Staff default: "Every correctness lease comes with a fencing token — in etcd, the create_revision of the lease-holder key; in ZooKeeper, the zxid or the sequential node number. The storage layer checks it. If the resource can't check tokens — say, a third-party API — the lock is best-effort, and I design the operation to be idempotent instead."

When to deviate:

• Efficiency-only locks (avoid duplicate cache warming): lease only; duplicates are harmless.
• Resource is a single database row: skip the external lock; use the row's version column as the fence.
• Hardware-backed bounded clocks (e.g., TrueTime-style uncertainty intervals): lease reads become safe inside the database because uncertainty is bounded and waited out — this is how Spanner's leaders serve reads — but only within that system.

🧭 Principal Move: "I'd split the platform API into efficiency_lock() and correctness_lease(). The second one requires the caller to declare the fenced resource in the service catalog. Design review rejects correctness claims on locks with no declared fence."

❌ Common L5 Trap: "Before each write, the worker checks it still holds the lock." The check and the write aren't atomic; a pause between them reopens the gap. The interviewer knows this — Martin Kleppmann's widely-read 2016 critique of Redlock makes exactly this point.

3.3 Fault Line 3: Linearizable Reads vs Read Scale#

The tension: Linearizable reads must confirm the leader is still leader (a quorum round) or trust a leader lease (clock assumptions). Serving from followers or caches scales reads ~N× but returns stale data — and can even go backwards across servers.

Staff default: "Correctness decisions use fencing or CAS, not reads. Humans and deploy tools use linearizable reads. Fleets use watches through a proxy with cached, revisioned state. Serializable reads for observability only."

When to deviate:

• ZooKeeper clients that must see their own writes elsewhere: sync() before read, or pin the session to one server (ZooKeeper guarantees per-session ordering).
• Very high read rates on a few keys: add learners in each zone; don't add voters.

🎯 Staff Insight: "The dangerous read is the one used to make a decision that then acts on a different system. By the time the decision lands, the read is history. That's why I fence rather than read."

3.4 Fault Line 4: Build vs Buy vs Avoid#

The tension: A custom consensus implementation can be tailored to your product's scaling limits. It is also one of the hardest correctness projects in software: edge cases in membership changes, snapshotting and log truncation surface years later.

Staff default: "Avoid if the resource has CAS. Otherwise run etcd — or use a managed offering if we're on one cloud. I'd only embed a Raft library if consensus is inside our product's data path, and I'd only build one if we're a database company."

When to deviate:

• You're building a database or a broker where metadata scale is the product limit (Kafka's KRaft): embed or build, with a team and a multi-year plan.
• Air-gapped or edge deployments where running etcd isn't possible: embed a library.

🧭 Principal Move: "Every coordination cluster in the company is a pager rotation, an upgrade backlog and a CVE surface. I'd count them. When I did this exercise the answer is usually 'more than the platform team knew about' — and the first project is consolidation, not new features."

3.5 Fault Line 5: Control Plane vs Data Plane#

The tension: It's simplest for services to consult the coordination service directly when they need a leader, a lock or a config value. That puts a majority-fsync system with 1–2s elections on the request path of every product.

Staff default: "Coordination is control plane. The data plane runs from cached state and keeps running — fail static — if coordination is unavailable. The only things that stop during an outage are changes: new leaders, new locks, new config. That's the right thing to lose."

When to deviate:

• Operations that are themselves control-plane (a deploy, a failover): they should block on coordination; better to wait than act on stale state.
• Security revocations (kill-switches, key revocation): may need a max-staleness bound (e.g., 30s) after which clients fail closed — a deliberate, signed-off exception.

🧭 Principal Move: "I'd write 'fail static' into the platform contract: every client library caches last-known-good, exposes its staleness as a metric, and keeps serving. Teams that want fail-closed must file an exception and own the resulting availability risk."

4. Failure Modes & Operational Reality#

4.1 Leader Flapping From a Slow Disk (Gray Failure)#

t=0:       Node 2 (leader) disk degrades: fsync p99 2ms → 450ms. Not dead.
t=+1s:     Heartbeats from leader delayed behind fsync. Followers hit election timeout (1s).
t=+1.2s:   Node 4 elected. Node 2 steps down. Writes fail for ~1.2s.
t=+20s:    Node 2, now a follower with less load, looks healthy; wins next election after node 4 hiccups.
t=+25s:    Node 2 leader again → flaps again.
t=+10min:  leader_changes_total = 38. Kubernetes API p99 12s. Controllers retrying. Deploys stall.
t=+12min:  Page: leader_changes > 3 in 15m. Runbook: identify node with worst wal_fsync p99.
t=+15min:  Node 2 removed from membership (member remove). Cluster stable on 4 voters.
t=+2h:     Replacement node added as learner, caught up, promoted.

Detection: etcd_server_leader_changes_seen_total, etcd_disk_wal_fsync_duration_seconds p99 per member, etcd_server_proposals_failed_total.

Mitigation: remove the sick member; don't just restart it.

Prevention: dedicated local SSD for WAL (not network storage shared with noisy neighbors); pre-vote and check-quorum enabled; election timeout ≥ 10× heartbeat and ≥ 5× p99 RTT; per-member disk latency alerts that fire before elections.

Owner: coordination platform on-call.

4.2 Split Brain at the Resource — The GC Pause#

t=0:       Billing scheduler A holds leadership (lease 10s, no fencing).
t=+3s:     A starts a full GC on a 64 GB heap: 24s pause.
t=+13s:    Lease expires. B elected. B starts nightly run.
t=+27s:    A resumes mid-loop, dispatches remaining 6,000 invoices.
t=+28s:    Both A and B dispatch. Duplicate invoices: 6,000.
t=+9h:     Customer complaints. Incident opened.

Detection: jobs.duplicate_dispatch_total (dedup at the downstream), JVM gc_pause_seconds_max, leader.stale_token_rejections_total (only exists once fencing is in).

Mitigation: credit notes; dedup on invoice idempotency key.

Prevention: fencing token on the job store; downstream idempotency; GC tuning to keep pauses well below lease TTL — but never rely on that alone.

Owner: the scheduler's team (resource owner) with the platform lock library team.

4.3 Quorum Loss During a Zone Outage + Maintenance#

t=0:       3-node cluster across zones A/B/C. Rolling OS patch: node C drained.
t=+5min:   Zone A has a power event. Node A down.
t=+5min:   1 of 3 voters alive. No quorum. All writes fail. Reads (serializable) still served.
t=+5min:   Leader elections impossible. Controllers can't update state. Deploys halt.
t=+25min:  Node C patch completes, rejoins. 2 of 3: quorum restored.

Detection: etcd_server_has_leader == 0, proposals_pending rising.

Mitigation: complete maintenance faster; never force a new cluster from one survivor unless data loss is accepted and signed off (it discards uncommitted entries and risks divergence).

Prevention: 5 voters for anything critical so maintenance of one node plus a zone loss is survivable; maintenance windows blocked during declared cloud-provider incidents.

Owner: platform on-call; change management owns the maintenance policy.

4.4 Watch / Reconnect Storm#

t=0:       Network blip 8s between the fleet and the coordination cluster.
t=+8s:     40,000 clients reconnect simultaneously; each re-establishes session + 5 watches.
t=+9s:     200,000 watch creations + full-state re-reads. Leader CPU 100%. Heartbeats late.
t=+11s:    Leader election triggered by overload. Clients see errors, reconnect again.
t=+30s:    Positive feedback loop — the storm keeps the cluster unhealthy.

Detection: watchers_count step changes, grpc_server_started_total rate, leader CPU.

Mitigation: proxy layer absorbs reconnects (1 upstream watch per key regardless of client count); clients reconnect with jittered backoff (0–30s) and resume watches from their last revision rather than re-reading everything.

Prevention: clients never connect directly at fleet scale; watch-resume from revision is mandatory in the client library.

Owner: platform (proxy and client library).

4.5 Database Size Exceeded — The Read-Only Cluster#

t=0:       A team starts storing 200 KB job specs as keys. 40,000 jobs/day.
t=+6 days: DB size 1.9 GiB of 2 GiB quota. No alert on size.
t=+7 days: NOSPACE alarm. Cluster rejects writes. Every tenant's elections and locks fail.

Detection: etcd_mvcc_db_total_size_in_bytes / quota > 70% warn, > 85% page.

Mitigation: compact old revisions, defragment member by member, disarm alarm; evict the offending tenant's keys.

Prevention: per-tenant byte quotas at the proxy; value size limit per tenant (e.g., 64 KB); scheduled compaction (etcd supports periodic auto-compaction).

Owner: platform; offending tenant owns the cleanup and migration of payloads to object storage.

4.6 Membership Change Gone Wrong#

Replacing a node by adding the new one before removing the old one briefly makes a 6-member cluster needing 4 for quorum; if the new node can't catch up (large snapshot, slow network), the cluster is now less available. Replacing two nodes at once can produce a moment where old and new configurations have disjoint majorities if the implementation doesn't use joint consensus or single-server changes.

Prevention: add as learner first, wait until caught up (lag < 1,000 entries), then promote; change one voter at a time; automate it — humans shouldn't type member add during an incident.

Owner: platform; runbook-automated.

4.7 Operational Reality Matrix#

🎯 Staff Insight: Most coordination outages are multi-tenant blast radius problems: one tenant's behavior (big values, many watches, write floods) takes down everyone's elections. Quotas at the access layer are not a nice-to-have.

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Reciting Raft's log-matching property ≠ designing a coordination service. Protocol fluency is table stakes; the level is decided at the boundaries.
• Mentioning CAP ≠ understanding tradeoffs. "It's CP" is a label. What matters is what becomes unavailable (writes, elections) and what keeps working (cached reads, the data plane).
• Mentioning Jepsen ≠ testing culture. The Staff answer names what they'd test: partitions during membership change, clock skew on lease holders, disk stalls on the leader.
• Big cluster sizes ≠ reliability. "Nine nodes for safety" usually signals someone who hasn't paid the write latency.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Paxos vs Raft proofs — one sentence: "equivalent safety, Raft's structure is easier to implement correctly."
• Byzantine fault tolerance — say it's out of scope unless nodes are mutually untrusted (blockchains, multi-party systems); BFT needs 3f+1 nodes.
• Log compaction internals — mention snapshots exist; skip formats.
• Transport details — gRPC vs custom; irrelevant to the design.

6.4 Follow-Up Questions to Expect#

1. "What's your fencing token, concretely, and who checks it?"
2. "Five nodes in two zones — what's your availability during a zone outage?"
3. "A client reads config from a follower, then reads from another follower and sees an older value. Is that allowed?"
4. "How do you replace a failed node without risking quorum?"
5. "One tenant is writing 5,000 keys/s. What happens to everyone else?"
6. "How long does failover take, and what's the tradeoff in tuning it faster?"
7. "When would you not use this service?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design a distributed coordination service like ZooKeeper."

Drill 2: Fencing — Make It Concrete#

Prompt: "You said fencing tokens. Where does the number come from and who checks it?"

Drill 3: Quorum Placement#

Prompt: "Your cloud region has three zones. How many nodes, where, and why?"

Drill 4: Coordination Is Down#

Prompt: "The coordination cluster has no quorum for 10 minutes. What happens to the company?"

Drill 5: The Hot Key / Herd Effect#

Prompt: "5,000 workers wait on the same lock. It's released. What happens?"

Drill 6: Multi-Tenant Platform#

Prompt: "30 teams want to use your coordination service. How do you keep one from hurting the rest?"

Drill 7: Build vs Buy#

Prompt: "Our database team wants to write their own Raft for a new storage engine. Do you approve?"

Drill 8: Changing Timeouts Safely#

Prompt: "Failover takes 3 seconds. Product wants 300ms. Change the election timeout?"

Drill 9: Multi-Region#

Prompt: "We're going to three regions. What happens to the coordination service?"

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Incident — Kubernetes Control Plane Meltdown During a Scale-Up#

Context: A marketing event triggers autoscaling from 800 to 3,000 nodes in 20 minutes. The Kubernetes API server p99 goes to 30s; pod scheduling stalls; the scale-up stalls with it. The on-call escalates to you.

Questions to Surface First:

• Is etcd the bottleneck (fsync, DB size, leader changes) or the API server (watch cache, request volume)?
• What's generating writes — node heartbeats, pod status updates, events?
• Is anything writing large objects?

Typical L5 Approach: Scales the API server replicas and increases etcd resources.

Staff Approach: Checks etcd first: wal_fsync_p99 is 40ms (network disk), and Event objects are 60% of writes. Moves events to a separate etcd cluster (a supported Kubernetes configuration), confirms node heartbeats use Lease objects rather than full status updates, and plans migration to local NVMe for WAL.

Principal Approach: Treats cluster size as a capacity-planned platform limit: publishes a per-cluster node ceiling (e.g., 2,000) and a multi-cluster strategy for larger footprints, and adds control-plane load tests to the pre-event readiness checklist.

Key Takeaway: "The coordination store's disk is the control plane's heartbeat. Pay for it."

What clears the Staff bar:

• Checks consensus-layer metrics before scaling stateless layers
• Separates write classes by criticality
• Turns an incident into a published capacity limit

Deep Dive 2: Silent Failure — The Lock That Never Protected Anything#

Context: An audit finds that a data-compaction job, "protected" by a ZooKeeper lock for two years, has occasionally run twice concurrently, causing 0.02% of partitions to have duplicated rows. Nobody noticed.

Questions to Surface First:

• Does the compaction writer check any token?
• When did concurrency happen — correlated with GC pauses, deploys, network blips?
• What else uses the same lock library?

Typical L5 Approach: Increases session timeout from 10s to 60s so the lock doesn't expire during pauses.

Staff Approach: Recognizes that longer timeouts only shrink the window and slow failover. Adds fencing: the compaction writer passes the lock node's sequence number to the storage layer, which rejects lower tokens per partition. Adds a detector — a nightly job counting duplicate rows — and backfills fixes.

Principal Approach: Audits every consumer of the lock library: 23 services, of which 17 claim correctness, 3 have a fence. Launches the correctness_lease API with required fence declaration and sets a 2-quarter deadline for the 14 unfenced users, tracked at the engineering-leadership level.

Key Takeaway: "A lock without a fence is a probability, not a guarantee."

What clears the Staff bar:

• Rejects timeout tuning as a fix for a safety problem
• Adds a metric that proves the fence is working
• Looks beyond the one job to the whole library's users

Deep Dive 3: Large-Customer Onboarding — The Team That Wants to Store 40 GB#

Context: A new ML platform team wants to use the shared etcd for model registry metadata: 2 million keys, 40 GB total, 3,000 writes/s during training bursts.

Questions to Surface First:

• What in this data actually needs consensus — probably "which model version is live", not every artifact's metadata?
• What are the read patterns and consistency needs?
• What happens to them if coordination is unavailable?

Typical L5 Approach: Increases the etcd quota to 64 GB and adds nodes.

Staff Approach: Declines: 40 GB exceeds practical etcd limits, would make snapshots and recovery take tens of minutes, and 3,000 writes/s would consume most of the shared cluster's budget. Proposes: metadata in PostgreSQL or DynamoDB; only the "live version pointer" per model (a few thousand keys) in coordination, with watches for rollout.

Principal Approach: Uses the request to publish the platform's acceptance criteria (size, rate, value size, use cases) and a decision guide — "coordination vs database vs cache" — so the next team self-selects correctly.

Key Takeaway: "Consensus is for agreement, not storage."

What clears the Staff bar:

• Separates the agreement-needing sliver from bulk data
• Protects other tenants' budget
• Turns a one-off no into a published policy

Deep Dive 4: Post-Mortem — Two Leaders After a Network Partition#

Context: During a partition, a primary database's failover manager promoted a replica while the old primary kept accepting writes for 90 seconds. 4,000 writes diverged. The failover manager used a consensus service for election.

Questions to Surface First:

• Did the old primary know it lost leadership? How would it find out?
• Did clients route writes by a cached "who's primary" value?
• Was there any fencing at the storage or proxy layer?

Typical L5 Approach: Shortens the old primary's lease check interval.

Staff Approach: Identifies that the consensus service elected correctly — the old primary simply couldn't reach it and kept serving. Fix: the old primary must stop accepting writes when it can't renew its lease within the lease period minus margin (self-demotion), and clients or proxies must route by epoch — writes carry the primary's epoch, and replicas/proxies reject writes from lower epochs. Divergent writes need reconciliation with owners.

Principal Approach: Standardizes failover across all database fleets on one manager with STONITH-style fencing (revoke network access or storage access of the old primary) and epochs, and requires a partition test in the failover certification of every database tier.

Key Takeaway: "Election is agreement among voters. Failover is making the loser stop. They're different problems."

What clears the Staff bar:

• Distinguishes a correct election from an incomplete failover
• Fences at multiple layers (self, proxy, network)
• Tests the exact partition shape that caused the incident

Deep Dive 5: Multi-Region Expansion — Global Leader for a Payments Scheduler#

Context: The company goes active-active in two regions. A payments settlement scheduler must have exactly one active instance globally. Someone proposes a 4-node etcd cluster, 2 per region.

Questions to Surface First:

• What happens when the inter-region link partitions?
• Can the scheduler be region-scoped instead (each region settles its own accounts)?
• What's the cost of a 10-minute scheduling pause vs a duplicate run?

Typical L5 Approach: Accepts the 4-node design; argues it's symmetric.

Staff Approach: Rejects 2+2: a partition leaves both sides with 2 of 4 — no majority anywhere, so no leader at all. Needs a third site: 5 voters as 2-2-1 with the 1 in a third region (or a lightweight witness location). Better: partition the work by region so each regional scheduler owns its accounts, eliminating the global leader; plus fencing via settlement-batch IDs in the ledger.

Principal Approach: Establishes that "global singleton" requirements need architecture review, since each one creates a three-region dependency; pushes the org toward region-partitioned ownership as the default pattern.

Key Takeaway: "Even voter counts across two sites buy you a partition with no leader. Always a third site."

What clears the Staff bar:

• Does the partition math on the proposed topology
• Removes the global singleton when possible
• Prefers unavailability over duplication for money

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Explain why consensus guarantees agreement inside the cluster but not exclusion outside it, and close the gap with fencing tokens
• Size and place a quorum against zones and regions, and state the write-latency cost of each shape
• Choose a read mode (fence, CAS, ReadIndex, watch, serializable) per use case
• Diagnose gray failures: slow disks, leader flapping, reconnect storms, quota exhaustion
• Perform membership changes safely: learner first, one voter at a time
• Argue build vs buy vs avoid with concrete criteria
• Design a multi-tenant coordination platform with quotas, tiers and fail-static clients

The Bar for This Question#

Mid-level (L4): Describes leader election and majority replication correctly. Uses a lock with a TTL. Doesn't address pauses, placement or blast radius.

Senior (L5): Explains Raft well, including terms and log matching; chooses 5 nodes; mentions ZooKeeper recipes and watches. The gap: treats the service's guarantees as the system's guarantees — no fencing at resources, node counts without failure domains, follower reads described as consistent, and a tendency to build rather than buy.

Staff+ (L6): Frames the service by intent and caps its scope. Fences at resources. Places voters against real failure domains and quantifies latency. Names gray failures with metrics and runbooks. Keeps coordination off the data plane with fail-static clients. Protects tenants from each other with quotas. Knows when a database's CAS makes the whole service unnecessary. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Distributed Locks" Are Mostly Misnamed Leases#

The Staff position: Call it a lease. Say whether it's for efficiency or correctness. If correctness, name the fence.

Why this matters in interviews: Precise language signals you've seen the failure. "Lock" without qualification signals you haven't.

10.2 Most Teams Asking for Consensus Need a Conditional Write#

The Staff position: Put mutual exclusion where the data is. External coordination is for when no single store owns the state.

Why this matters in interviews: Reducing scope is a senior judgment signal that most candidates never show.

10.3 Seven Nodes Is Almost Always Wrong#

The Staff position: Real failure domains come in threes (zones, regions). Five voters across three domains covers the realistic double fault. Seven adds write latency and operational surface for failures that are rarely independent anyway.

Why this matters in interviews: Picking cluster size from failure domains, not from "more is safer", shows operational maturity.

10.4 The Coordination Service Is Your Biggest Single Point of Failure — Even Though It Has No Single Point of Failure#

The Staff position: Replication protects against independent failures. Tiering, quotas, fail-static clients and staged upgrades protect against correlated ones — which are the ones that happen.

Why this matters in interviews: It reframes availability from nodes to dependencies — an L6+ lens.

10.5 You Should Never Write Your Own Consensus — Until It's Your Product#

The Staff position: Every mature system that built its own (Chubby, ZooKeeper, etcd, KRaft) had a team whose job was that system, for years. If coordination isn't your product, it isn't your project.

Why this matters in interviews: "I'd run etcd" is not a cop-out; it's the answer — provided you then show you understand what etcd can't do for you.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

The Staff engineer designs a correct, well-placed coordination cluster. The Principal engineer finds that the company runs eleven of them — three ZooKeeper ensembles from the Kafka and Hadoop era, five etcd clusters for Kubernetes, a Consul cluster a team adopted for service discovery, and two Redis "lock services" nobody admits to — each with different upgrade states, different on-call owners and different ideas of what "lock" means. The L7 problem is coordination as a governed platform: how many clusters, which products, which guarantees, who is on call, and how the organization avoids a correlated failure that takes down every control plane at once.

The Org-Level Fault Line#

One shared coordination platform vs per-team clusters.

🧭 Principal Move: "One product, one owner, many clusters. The platform team runs a fleet of identical, tiered clusters — tier-0 dedicated for Kubernetes control planes, tier-1 shared with quotas for services — and upgrades them in waves, never all at once. Teams get isolation without owning consensus."

Cost Model#

Assumptions: cloud VMs with local NVMe; fully loaded engineer ~$250K/year; figures are order-of-magnitude.

The pricing insight: the dominant cost is never the VMs — it's engineer time on upgrades, incidents and bespoke clusters. Consolidating 11 unmanaged clusters into a fleet typically saves more in on-call and upgrade toil than it costs in platform headcount within a year.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-COORD-001: Coordination & Locking Standard
Status: Approved   Owner: Coordination Platform

Scope
  Any use of consensus-backed coordination (leader election, locks, leases,
  membership, configuration) by production services.

MUST
  1. Use the platform coordination service or a platform-approved managed equivalent.
     No new self-managed ZooKeeper/etcd/Consul clusters.
  2. Classify every lock as EFFICIENCY or CORRECTNESS.
  3. CORRECTNESS leases MUST pass a fencing token to a declared resource that rejects
     stale tokens; the resource is registered in the service catalog.
  4. Data-plane clients MUST fail static on cached state and expose staleness as a metric.
  5. Stay within tenant quotas: ≤ 50 MB, ≤ 64 KB per value, ≤ 200 writes/s unless approved.

SHOULD
  1. Prefer resource-native conditional writes over external locks.
  2. Use watches via the platform proxy, resuming from the last revision.
  3. Keep coordination out of synchronous user request paths.

Exceptions
  Architecture review; region-spread (global) coordination requires a latency sign-off
  from the consuming service's owner and director.

Success metrics
  - Unmanaged coordination clusters: 0 by end of year 2
  - CORRECTNESS leases without a declared fence: 0
  - Coordination-caused user-facing incidents: ≤ 1 per year
  - Fleet upgrade lag: all clusters within 1 minor version within 60 days of release

What I'd Tell the VP#

"Every deploy, scheduler and failover in the company depends on a handful of small coordination clusters — and today we run eleven of them with six different owners, two of which nobody is on call for. The risk isn't that one fails; it's that a bad upgrade or a misbehaving team takes several down at once and stops our ability to deploy the fix. I'm proposing one platform team that runs a standardized fleet with per-team limits, and a lock standard that closes the class of 'two workers did the same job' incidents we've had three times this year. It's about four engineers; it retires roughly the equivalent of two engineers' worth of scattered toil in year one and removes a correlated-outage risk we can't currently quantify."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll run a well-configured 5-node etcd" — correct for one cluster, silent on the ten others and on tenancy.
• "Teams should use fencing tokens" — right principle, no mechanism to make it happen across 30 teams.
• "We'll monitor leader changes and fsync latency" — good operations, no answer for correlated upgrade risk or deploy break-glass.

Appendices

Appendix A: Raft Mechanics in Depth#

A.1 Roles and Elections#

• Each node's timeout is randomized (e.g., 150–300ms in the paper; ~1s in etcd defaults) so split votes are rare.
• A node votes at most once per term, and only for a candidate whose log is at least as up-to-date — the election restriction that guarantees a new leader has every committed entry.
• Pre-vote (etcd, others): a would-be candidate first asks whether it could win before incrementing its term, so a partitioned node rejoining doesn't disrupt a healthy leader.
• Check-quorum: a leader that can't hear from a majority steps down, limiting how long a partitioned leader believes it leads.

A.2 Log Replication#

Commit cost = leader fsync ∥ (network RTT + follower fsync) for the fastest majority. On NVMe within a region: ~1–4ms.

A.3 Membership Changes#

Change one voter at a time (single-server changes) or use joint consensus; never add and remove multiple voters in one step. Removing the failed node first (when the rest is healthy) keeps quorum requirements lower during catch-up.

A.4 Why Paxos and Raft Are "the Same" for Interviews#

Both need a majority, both use a monotonically increasing ballot/term, both guarantee that once a value is chosen no conflicting value can be chosen. Raft packages leader election, log replication and membership into a prescribed structure; Multi-Paxos leaves them as engineering choices. Choose by implementation maturity, not by algorithm.

Appendix B: Keys, Recipes and Data Model#

/tenants/{tenant}/elections/{name}/{lease_id}     # leader = lowest create_revision
/tenants/{tenant}/locks/{name}/{lease_id}          # waiters watch predecessor
/tenants/{tenant}/members/{service}/{instance}     # attached to instance lease
/tenants/{tenant}/config/{component}               # watched; value ≤ 64 KB

Leader election recipe (etcd-style):

lease = LeaseGrant(ttl=10s); keepalive(lease) every 3s
Put(prefix + lease.id, my_addr, lease=lease)
loop:
  kvs = Get(prefix, sort=create_revision asc)
  if kvs[0].key == prefix + lease.id:
      token = kvs[0].create_revision     # fencing token
      act_as_leader(token)               # every write carries token
  else:
      Watch(kvs[my_index-1].key) until deleted

Resource-side fence:

UPDATE job_runs SET state = $2, leader_token = $3
 WHERE job_id = $1 AND leader_token <= $3;
-- 0 rows → stale leader → exit

Appendix C: Coordination Mechanisms — Quick Comparison#

Appendix D: Client Contract & Behavior#

• Sessions: keepalive every TTL/3; on missed keepalives, the client assumes loss at TTL − margin by its monotonic clock and stops acting as leader before the server expires it.
• Reconnect: exponential backoff with full jitter, base 100ms, cap 30s; resume watches from last revision; never re-read full state unless the revision was compacted (then re-list once).
• Caching: last-known-good persisted locally; staleness exposed as coord.client.staleness_seconds.
• Errors: distinguish NO_LEADER (retry later), QUOTA_EXCEEDED (tenant bug, don't retry fast), COMPACTED (re-list), STALE_TOKEN from resources (step down, never retry).

Appendix E: Observability#

Core metrics (etcd names where applicable):

• etcd_server_has_leader, etcd_server_leader_changes_seen_total
• etcd_disk_wal_fsync_duration_seconds p99, etcd_disk_backend_commit_duration_seconds p99
• etcd_server_proposals_failed_total, etcd_server_proposals_pending
• etcd_mvcc_db_total_size_in_bytes vs quota
• etcd_network_peer_round_trip_time_seconds
• Proxy: watchers_count, per-tenant write_rate, bytes, throttled_total
• Clients: coord.client.staleness_seconds, lock.stale_token_rejections_total

Critical alerts:

Control plane vs data plane: coordination is control plane; its outage should appear in deploy latency and controller lag, not in user error rates. A user-error-rate correlation with coordination health is itself an alertable design violation.

Appendix F: Scale Evolution#

What you don't build on day one: a custom consensus implementation, a global cross-region cluster, a proxy layer (until ~1K clients), chargeback.

Appendix G: Multi-Tenancy, Fairness & Cost#

• Quotas at the proxy, not the core: the consensus core has no notion of tenants; enforcement must happen before proposals are created.
• Weighted write budgets: tier-0 tenants reserve 50% of tested write capacity; tier-1 share the rest with per-tenant caps.
• Watch fairness: coalesce identical watches; cap per-tenant watch count (e.g., 5,000 upstream) — a tenant with 50,000 unique watches is a design review.
• Showback: report per-tenant writes/day and bytes; the conversation "your team is 60% of our write load" resolves most noisy-neighbor issues without enforcement.