etcd vs ZooKeeper vs Consul

All three are small, replicated, strongly consistent stores for a few megabytes of critical metadata, so they get compared as interchangeable "coordination services". They are not interchangeable in what they are for. etcd is a consistent key-value store with watches and leases, built as the substrate for control planes like Kubernetes. ZooKeeper is the original coordination kernel, a tree of znodes with sessions and ephemeral nodes that a generation of data systems embedded. Consul is a service networking product — service catalog, health checks, DNS, a KV store and a mesh — with consensus inside it and gossip around it. The question that decides it is: are you building a control plane that needs a consistent store, or do you need service discovery as a product? A consistent store points to etcd (or the one your platform already runs). Discovery and health checking across VMs and datacenters points to Consul. ZooKeeper is the answer mainly when the software you are running already requires it.

The Verdict#

Default: etcd for new coordination needs, and only if a conditional write in your existing database will not do. Consul when the requirement is service discovery and health checking outside Kubernetes. ZooKeeper when a system you depend on still requires it.

🎯 Staff Move: "For leader election and the shard map I'd use etcd with a 10-second lease and fencing tokens — or, honestly, a conditional write in Postgres, since we already run it. Service discovery is a different problem: it wants availability during partitions, so I'd use the platform's discovery, not a hand-rolled registry in a consensus store."

When the non-default wins:

• ZooKeeper for new work wins only when you are extending a system that already runs it well and has client recipes battle-tested against it; adding a second consensus system to avoid it is worse.
• Consul for config and locks wins when Consul is already deployed for discovery on every node: one fewer system beats a slightly better fit.
• None of them wins more often than people expect. Leader election for a handful of singleton jobs works with a conditional update on a lease row in the database you already run, with the row's epoch as the fencing token.

At a Glance#

Numbers to bring:

How They Actually Differ#

What all three share, so you can skip it in the interview:

• A leader-based consensus protocol with majority quorums: writes go through one leader and are fsynced on a majority before acknowledgment.
• The whole dataset is meant to be small and largely memory-resident; they are metadata stores, not databases.
• Liveness by heartbeat (lease, session or health check), which means pauses longer than the timeout look like death.
• Watches or blocking queries instead of polling, and a monotonically increasing version number usable as a fencing token.

1. Store vs Product#

etcd and ZooKeeper are building blocks: they give you primitives — compare-and-swap, leases or sessions, watches, ordered writes — and you build leader election, locks or membership on top, ideally with a well-tested client library recipe. Consul is a product with an opinion: a node runs an agent, services register with health checks, and other services find healthy instances through DNS or HTTP. The KV store and locks are there, but they are secondary.

That changes who owns what. With etcd you own the recipes and the correctness of every client. With Consul you own an agent on every node and a product's upgrade cadence, but discovery and health checking work without writing code.

2. Reads: Who Answers and How Fresh#

This is the most interview-relevant difference. ZooKeeper's local reads make it fast and read-scalable but surprise people who assumed linearizability: a client can read a value that another client has already seen overwritten, unless it calls sync first. etcd made the opposite default. Consul chose a split default that matches its purpose: catalog lookups through DNS favor availability, and API reads favor consistency.

🎯 Staff Insight: Service discovery is an availability problem wearing a consistency costume. A slightly stale list of healthy instances is fine; no list at all during a leader election is an outage. That is why Consul's DNS reads are stale by default — and why putting discovery on linearizable reads is the wrong instinct.

3. Liveness: Leases, Sessions and Gossip#

• etcd leases: the client creates a lease with a TTL (commonly 10–15s for leader election) and keeps it alive; keys attached to the lease vanish when it expires. One lease can cover many keys, so 1,000 workers cost 1,000 keepalives, not one per key.
• ZooKeeper sessions: the client holds a session with a negotiated timeout (commonly 6–30s); ephemeral znodes die with the session. A long JVM GC pause on the client expires its session and drops its locks and registrations — the classic ZooKeeper production incident.
• Consul: liveness of services comes from health checks run by the local agent (HTTP, TCP, script, TTL), plus gossip-based failure detection between agents with suspicion and refutation to avoid flapping. This scales to thousands of nodes without each node heartbeating the servers, which is the main reason Consul handles large fleets better than a registry built on raw sessions or leases.

The fencing step is identical for all three systems: the lock service cannot stop a paused ex-leader, only the resource can.

4. Scale and Topology#

All three have the same consensus arithmetic: 3 voters tolerate 1 failure, 5 tolerate 2, even numbers add latency without adding tolerance, and adding voters slows writes. They differ in how they scale everything else.

Consul's multi-datacenter model is the clearest built-in story of the three: each datacenter has its own Raft cluster, so losing a WAN link does not stop local writes, and queries can be forwarded across datacenters when needed.

Where Each One Breaks#

The production surprise for each:

• etcd: nobody watches the database size until the NOSPACE alarm stops the control plane on a Friday.
• ZooKeeper: the incident is usually caused by a client GC pause, not the ensemble.
• Consul: it becomes the dependency of everything — discovery, secrets bootstrap, config, deploys — so its outage is a company outage. Blast radius is the design problem.

Incident Sketch: The Session Expiry Storm#

t=0        Top-of-rack switch flaps for 9 seconds in one zone
t=+6s      ZooKeeper sessions with a 6s timeout expire for 400 workers in that zone
t=+7s      Their ephemeral znodes vanish; the coordinator sees 400 departures
t=+8s      Coordinator reassigns 3,000 partitions to the remaining zones
t=+12s     Network recovers; 400 workers reconnect with new sessions and rejoin
t=+13s     Coordinator reassigns again; two full rebalances in 10 seconds
t=+5min    Consumers still catching up; downstream lag alarms

Detection: session expirations per minute, rebalance count, membership churn. Mitigation: longer session timeouts for members whose loss is cheap to delay, a grace period before reassigning work, and rebalance rate limiting. Owner: the client team that wrote the membership recipe, with platform setting the defaults.

Incident Sketch: etcd Hits Its Quota#

t=0        A new controller writes a status key per object every 10 seconds
t=+5d      etcd database grows from 600 MB to 2 GB; compaction keeps revisions, defrag never scheduled
t=+5d+1m   NOSPACE alarm: etcd rejects writes
t=+5d+2m   Control plane cannot create, update or delete objects; running workloads continue
t=+5d+40m  Compact, defragment member by member, disarm alarm; offending controller rate-limited

Detection that would have caught it: database size as a percentage of quota with an alert at 60%, writes per second by client, and keys per prefix. Prevention: per-client write budgets, scheduled defragmentation, and a review for any controller that writes status at a fixed interval.

Cost and Operations#

The infrastructure bill is trivial for all three — a few hundred to a few thousand dollars a month. The cost is on-call and expertise. A consensus cluster is a tier-0 dependency that needs someone who understands quorum loss at 3 a.m. Running two different ones doubles that.

Assumptions: small VMs with SSDs at cloud list prices; the dominant cost at every tier is expertise and on-call.

🧭 Principal Insight: The org-level question is how many consensus systems you run, not which one is best. Each extra one is another tier-0 dependency with its own failure modes, upgrade path and expertise. Standardize on one per use case and retire the rest.

Switching Later#

Retiring a ZooKeeper-based recipe, in order:

1. Put the recipe behind a small internal interface (elect(), heartbeat(), currentLeader()) in every client.
2. Implement the interface on the new backend with the same fencing token semantics.
3. Run both for one role at a time, with the new backend authoritative only after a clean week.
4. Remove the old backend per role, then decommission the ensemble once no client connects.

The one-way doors:

1. Agents on every node (Consul). Once discovery, health checks and mesh depend on them, removing the agent is a fleet-wide migration.
2. Hand-written recipes against a specific API. Leader election code that depends on ZooKeeper's sequential ephemeral znodes does not port; code that depends on a small internal interface does.
3. Putting application data in the coordination store. Getting it out later means finding every reader, and the store's size problems arrive first.

How Real Companies Chose#

Kubernetes — etcd as the Single Backing Store#

Kubernetes keeps all cluster data in etcd, which its documentation describes as a consistent and highly available key-value store and the backing store for all cluster data. It recommends an odd number of members, five for production, dedicated machines or isolated environments, and a backup plan, and warns that disk and network starvation cause heartbeat timeouts and instability — without a leader, no cluster state can change (Kubernetes docs).

Staff insight: etcd's flagship user runs it as a dedicated, five-member, backed-up tier-0 store behind a caching API server. That is the template for any control plane you design.

Apache Kafka — Replacing ZooKeeper with a Built-In Quorum#

KIP-500 replaced Kafka's ZooKeeper dependency with a self-managed, Raft-based metadata quorum. The stated reasons: operators had to learn and secure two distributed systems; metadata held outside Kafka could diverge from controller state; and controller failover required reloading all state, while a quorum of controllers that already track the metadata log can fail over quickly and support far more partitions (KIP-500).

Staff insight: For a product, an external coordination service is a tax on every operator. If you are building the product, embed consensus; if you are buying products, prefer the ones that already did. ClickHouse made a similar move with ClickHouse Keeper, a Raft-based, ZooKeeper-protocol-compatible replacement with linearizable reads (ClickHouse docs).

Roblox — When Consul Became a Single Point of Failure#

Roblox's 73-hour outage in October 2021 centered on Consul, which served service discovery for thousands of services and underpinned Nomad and Vault. Two causes combined: a newly enabled streaming feature created contention under heavy concurrent read and write load, and a BoltDB freelist pathology made each Raft log append rewrite megabytes of bookkeeping. Afterward Roblox split critical workloads into dedicated Consul clusters, moved inappropriate data out of the KV store, removed circular dependencies between telemetry and Consul, and moved to bbolt (Roblox postmortem).

Staff insight: The product choice was not the root problem; the blast radius was. One coordination cluster underneath discovery, scheduling, secrets and telemetry turns any bug into a total outage. Cells, separate clusters per function, and break-glass paths are the design.

Follow-Ups to Expect#

What to Say in the Interview#

"These are metadata stores, not databases. I'll keep data in them to kilobytes per key and a few hundred megabytes total, and keep them off the per-request path so losing quorum freezes changes but not traffic."

"For a new control plane I'd pick etcd because reads are linearizable by default and it's what Kubernetes already runs. For discovery across VMs and datacenters I'd pick Consul, because discovery needs to keep answering during a partition."

"Every lock is a lease, and every lease needs a fencing token checked by the resource — that's true whichever of the three we choose."

"The bigger risk is blast radius: if discovery, secrets and deploys all sit on one cluster, its bad day is our outage. I'd separate clusters by function and keep a break-glass path."

Related Guides#

• etcd and ZooKeeper — internals, recipes, fencing and failure modes in depth
• Design a Consensus Service — building the store itself
• Design a Service Registry — discovery as an availability problem
• Design Distributed Locking — leases, fencing and when not to lock
• Design a Job Scheduler — singleton workers and leader election in practice
• Kubernetes — the canonical control-plane / data-plane split on etcd
• Coordination Strategies — choosing between consensus, conditional writes and avoidance
• Feature Flags — config distribution with watches and staged rollout