Technologies referenced in this case study: ZooKeeper & etcd · API Gateways & Service Mesh · Redis

Related case studies: Load Balancer · Circuit Breakers · API Gateway · Distributed Consensus · Distributed Coordination · Consistent Hashing

How to Use This Case Study#

Organized for interview use first, reference second.

Without a well-designed discovery system:
t=0:      Rolling deploy of payments-svc begins; old pods terminate as new ones start
t=+5s:    Old pods are killed immediately; registry still lists them (TTL 90s)
t=+10s:   Callers cached the endpoint list 30s ago; ~40% of their endpoints are dead
t=+15s:   Connection-refused and timeouts; callers retry into other dead endpoints
t=+90s:   Registry finally expires old entries; callers refresh over the next 30s
t=+2min:  Deploy "succeeded" — and payments had a 2-minute 35% error rate

With discovery designed for churn:
t=0:      Same deploy; pod receives SIGTERM → fails readiness → deregistered in <2s
t=+2s:    Control plane pushes updated endpoints to all proxies (p99 < 3s)
t=+5s:    Pod drains in-flight requests for 20s, then exits
t=+5s:    Callers' proxies already stopped sending new requests to it
Result:   Zero-error deploy; the same churn, handled as a routine event

Executive Summary

If you only read one section, read this.

What This Interview Actually Tests#

Service discovery is not a key-value lookup question. Everyone can draw "service registers in ZooKeeper, client reads ZooKeeper."

This is a control-plane availability and freshness question that tests:

• Whether you know the registry must be more available than every service it describes — and what that implies for CP vs AP
• Whether you separate the control plane (who knows the truth) from the data plane (who routes requests) so one can fail without the other
• Whether you reason about who decides an instance is healthy — and what happens when that judgment is wrong for the whole fleet at once
• Whether you design for churn — deploys, autoscaling, preemption — as the normal case, not the exception

The key insight: Discovery data is eventually consistent no matter what you pick, because the world changes faster than any registry can observe it. The design question is how stale a caller's view may get, what it does when the view is wrong, and whether a registry outage can ever take down traffic that was working fine.

The L5 vs L6 vs L7 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll design for intra-cluster RPC routing — that's the high-churn, high-fan-out case where the hardest tradeoffs live. I'll call out that leader and config discovery are a different problem that needs a consensus store with fencing, and that I would not put both on the same system with the same availability posture."

The Five Fault Lines#

In the Wild: Real Production Systems#

Netflix — Eureka and the Case for AP#

Netflix built Eureka for AWS, where instances come and go and network partitions between availability zones are real. Instances renew a lease every 30 seconds; entries expire after 90 seconds without renewal; clients cache the full registry and refresh every 30 seconds. Eureka servers replicate peer-to-peer without consensus. Its signature feature is self-preservation: if renewals across the fleet drop below ~85% of expected, the server assumes it is partitioned, not that the fleet died, and stops expiring instances.

Staff insight: Eureka is the explicit argument that discovery should prefer availability. Self-preservation is a panic threshold at the registry level — "if too much looks dead at once, the observer is probably wrong."

Airbnb — SmartStack (Nerve + Synapse)#

Airbnb's SmartStack (open-sourced ~2013) put a small health-checking agent (Nerve) next to each service instance, registering it in ZooKeeper only while healthy, and a local agent (Synapse) on each caller host that watched ZooKeeper and rewrote a local HAProxy config. Applications just called localhost:port. If ZooKeeper became unavailable, HAProxy kept routing with its current config.

Staff insight: SmartStack is a sidecar service mesh before the term existed: language-agnostic, local proxy, fail-static by construction. The pattern — control plane watches a registry, data plane is a local proxy — is what Envoy and xDS later standardized.

Kubernetes + Envoy/xDS — Platform-Registered, Push-Based#

In Kubernetes, services don't register themselves: the EndpointSlice controller derives endpoints from pods whose readiness probes pass (EndpointSlices hold up to 100 endpoints each by default, to keep update fan-out small). kube-proxy or a mesh control plane (e.g., Istio's istiod) watches those objects and pushes endpoints to proxies over xDS (EDS for endpoints). Envoy, created at Lyft, keeps serving with its last-received config if the control plane disappears, and applies a panic threshold (default 50%): if fewer than half of hosts are healthy, it load-balances across all of them.

Staff insight: This is the modern reference design: orchestrator is the source of truth, readiness is the service team's contract, push-based distribution with incremental updates, and a data plane that fails static.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: Truth flows one way — orchestrator → registry → controller → discovery servers → proxies — and requests never flow through the control plane. Each sidecar holds a last-known-good endpoint list, so if etcd loses quorum or every discovery server crashes, traffic continues on the current view; only changes stop propagating. Pod 2 is draining: it failed readiness, was removed from endpoints, and is finishing in-flight requests before exiting. DNS serves clients that can't speak xDS, at worse freshness. The key metrics are convergence time and the age of each proxy's config.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The prompt usually arrives as "Design service discovery for a microservices platform", "How do services find each other?", or "Design something like Consul." The trap is to spend the interview on the registry's data model. The registry is the easy part. The hard parts are propagation under churn, health semantics, and what happens when the control plane fails.

Phase 1: Requirements & Framing (2–3 min)#

"A few questions before I draw anything. How big is the fleet — hundreds of services or thousands, and how many instances? How much churn — are we autoscaling and deploying dozens of times a day? How many languages do callers use? And is this only for RPC routing, or also for things like leader election and config? I'll assume 1,500 services, 60,000 instances, 3 clusters across 2 regions, heavy autoscaling so ~500 endpoint changes per minute at peak, 4 languages, and RPC routing as the primary use. Leader election I'll treat as a separate problem."

What you've established:

• Scale and churn set the propagation design
• Languages decide client-side vs sidecar
• Scope separates endpoint discovery (AP-friendly) from coordination (CP-required)

Phase 2: Core Entities & API (1–2 min)#

API — the control plane's contract, not a per-request lookup:

Registration (platform-internal; not called by services):
  UpsertEndpoint(service, endpoint, ready, zone, metadata)
  RemoveEndpoint(service, endpoint)

Subscription (proxies / proxyless clients):
  Watch(services[], known_versions{}) → stream of Delta{service, added[], removed[], changed[], version}
  Ack(version) / Nack(version, error)       # client rejected a bad config

Fallback:
  DNS: payments.svc.cluster.local → A/AAAA or SRV records, TTL 30s

🎯 Staff Move: "Notice there's no 'Lookup(service)' on the request path. Callers subscribe once and receive deltas. Every request is routed from local memory. That's the property that makes a registry outage survivable."

Phase 3: High-Level Architecture (≤5 min)#

Draw the overview diagram and name five components with owners:

1. Source of truth — the orchestrator (scheduler + node agent readiness). Owner: platform/compute.
2. Registry store — consensus-backed (etcd/Raft), 3–5 nodes per cluster. Owner: platform.
3. Discovery servers — stateless, horizontally scaled, watch the store and push deltas to subscribers. Owner: platform/networking.
4. Data plane — sidecars or proxyless clients: local endpoint cache, load balancing, outlier ejection, fail static. Owner: platform (mechanics), service teams (per-service overrides).
5. DNS fallback — for legacy and third-party clients. Owner: platform/networking.

"The registry is the least interesting box. The interesting questions are how fast changes reach callers, what happens when health signals are wrong, and what the data plane does when the control plane is gone."

Phase 4: Transition to Depth#

"I'd like to go deep on three things, in order of how often they cause real incidents: deploy-time churn and convergence, health signals and mass deregistration, and control-plane failure — including the fan-out problem when thousands of proxies reconnect at once. Then ownership and multi-cluster."

Phase 5: Deep Dives (25–30 min)#

Deep dive 1 — Convergence under churn (7 min).

Target: an endpoint removed from service must stop receiving new requests within 5s (p99)
Pipeline budget:
  readiness failure detected       ≤ 2s   (probe period 1s, failureThreshold 2 during shutdown via preStop)
  controller updates slice          ≤ 0.5s
  discovery server debounce         100ms
  push to 10,000 subscribers        ≤ 2s  (incremental delta, only subscribers of this service)
  total                             ≈ 4.6s  → drain wait = 5s + in-flight max (e.g. 10s) → grace period 20s
Graceful shutdown: SIGTERM → preStop: fail readiness, sleep 5s → stop accepting → drain 10s → exit

Deep dive 2 — Health and mass deregistration (7 min). Three layers: orchestrator readiness (service-defined), caller-side outlier ejection (e.g., 5 consecutive 5xx → eject 30s, max 10%), and panic threshold (< 50% healthy → route to all). Add a control-plane guard: if a single update would remove > 30% of a service's endpoints, delay and alert rather than apply instantly.

Deep dive 3 — Control-plane failure and fan-out (7 min). If etcd loses quorum, no updates propagate; data plane keeps last-known-good. Endpoints that die during the outage are handled by outlier ejection and retries. When the control plane recovers, 10,000 proxies reconnect — add jittered reconnect backoff, resume from known versions (send deltas, not full state), and rate-limit full pushes.

Deep dive 4 — Scoping fan-out (4 min). A proxy that subscribes to every service receives every change: 1,500 services × churn → most updates are irrelevant. Scope subscriptions to declared dependencies (Istio's Sidecar resource is a real example). Push cost drops from O(proxies × changes) to O(interested proxies × changes).

Deep dive 5 — Multi-cluster (3 min). Each cluster is authoritative for its own endpoints. Cross-cluster discovery is federated read-only; routing prefers local zone → local region → remote with capacity caps.

Phase 6: Wrap-Up (2–3 min)#

"To summarize: the orchestrator is the source of truth, a consensus store holds endpoints per cluster, discovery servers push scoped deltas to sidecars and proxyless clients, and the data plane fails static with outlier ejection and a panic threshold. Deploys drain after deregistration using a wait derived from convergence p99. What I'd build next: convergence SLOs per cluster, a mass-deregistration guard, and federation with locality-aware failover. The biggest remaining risk is the control plane as a shared failure domain — I'd cell it per cluster and test 'control plane dead for 1 hour' quarterly."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Discovery is the ultimate shared dependency. Every service depends on it; no service team owns it; and its failure modes are correlated — when it breaks, it breaks for everyone at once. That makes it a perfect Staff question: the technical design is modest, but the consequences of each decision are fleet-wide. A Senior engineer designs a registry. A Staff engineer designs a registry whose worst day doesn't take down the company.

It also forces precision about CAP. Candidates who say "we need strong consistency" are usually right about the registry and wrong about the system: the data plane must be available even when the registry isn't.

1.2 The L5 vs L6 vs L7 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"If the registry disappeared for an hour right now, what would stop working?"

The correct answer is: "Nothing that's currently working. New deploys and scale-ups wouldn't become routable, and instances that die would be caught by outlier ejection instead of the registry — but no existing request path depends on the registry being reachable." If your design can't say that, redesign it until it can.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Intent 1: Intra-cluster RPC routing. High churn, high fan-out, latency-sensitive. Staleness of a few seconds is fine because retries and outlier ejection cover dead endpoints. What's not fine: a registry outage that blocks routing, or a health bug that empties a service. Design for availability of the data plane and bounded convergence.

Intent 2: Cross-cluster and multi-region discovery. The new problem is that clusters fail independently and WAN links partition. Each cluster should be authoritative for its own endpoints, with other clusters' endpoints federated read-only. The danger is failover: if cluster A's payments dies and every caller in A starts calling cluster B's payments, B receives 2× load. Locality-aware routing with capacity caps.

Intent 3: Coordination discovery. "Who is the leader of shard 7?" "Which instance owns the cron scheduler?" These need linearizable answers and fencing, not eventual consistency — two leaders means corrupted data. Use a consensus store directly (leases in etcd, ephemeral sequential nodes in ZooKeeper) and fencing tokens; see Distributed Coordination.

🎯 Staff Move: "Endpoint discovery tolerates staleness and must be available; leader discovery tolerates unavailability and must be correct. Putting both behind one API with one failure posture is how you get either a fleet outage or a split brain."

2.2 When NOT to Build (or Use) a Discovery System#

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: CP vs AP Registry#

The Staff default: CP store for the write path (etcd behind the orchestrator), strictly cached data plane that fails static. You get a single authoritative history and an available routing layer. The CAP choice is made separately for the two planes.

🎯 Staff Move: "I'm happy with a CP registry, as long as nothing in the request path reads it. The CAP decision I care about is the data plane's, and there I choose availability: route on the last good view."

When to deviate: Environments without an orchestrator and with frequent partitions between zones (classic early-cloud VMs) — an AP registry like Eureka with self-preservation is reasonable. Say so, and note the cost: longer ghost-endpoint windows.

3.2 Fault Line 2: Client-Side vs Server-Side Discovery#

The Staff default: Sidecar for the general fleet (polyglot), proxyless xDS for the highest-QPS services where the sidecar tax is measurable, and DNS/VIP for legacy clients. One control plane feeds all three.

When to deviate: A mostly single-language shop (e.g., all JVM) with < 100 services can run a good client library and skip the mesh. Name the trigger for revisiting: the second language, or the first incident caused by library version skew.

3.3 Fault Line 3: Push vs Pull Propagation#

Fan-out math, full-state push:
  10,000 proxies subscribed to all services
  payments has 2,000 endpoints (~100KB of endpoint data)
  deploy changes payments ~40 times per minute (batched)
  full push: 10,000 × 100KB × 40/min ≈ 40 GB/min from the control plane
Scoped + incremental:
  800 proxies actually call payments; delta ~2KB per batch
  800 × 2KB × 40/min ≈ 64 MB/min — ~600× less

The Staff default: Push, incremental, scoped to declared dependencies, with 100ms–1s debounce to batch changes during deploys. DNS stays as the fallback for clients that can't subscribe.

🎯 Staff Move: "Push is only better than poll if it's scoped and incremental. Full-state push to every proxy is how control planes melt during the biggest deploy of the day."

3.4 Fault Line 4: Who Decides Health#

The critical failure: readiness probes that check dependencies. If every payments pod's readiness checks the payments DB and the DB blips, every pod goes unready simultaneously, the service has zero endpoints, and a 5-second DB blip becomes a full outage that lasts until probes pass again (plus propagation). Readiness should check this instance's ability to serve, not shared dependencies.

The Staff default: Orchestrator readiness as the registry's health truth (service-owned, checks only local state), caller-side outlier detection for fast per-caller reaction (capped at 10% ejection), panic threshold at 50%, and a control-plane guard against removing > 30% of a service's endpoints in one update.

3.5 Fault Line 5: Fail Static vs Fail Accurate#

When the data plane can't reach the control plane, or the control plane sends something suspicious, what happens?

The Staff default: Fail static without expiry for endpoints (combined with outlier ejection to cover dead hosts), validate every push (NACK suspicious updates such as an empty endpoint set for a service that had 200), and alert on stale_config_age_seconds > 60. Persist last-known-good to local disk so proxy restarts during a control-plane outage don't come up empty.

🎯 Staff Move: "Stale data with outlier ejection routes most requests correctly. No data routes none. I'll take stale."

When to deviate: Security-sensitive routing (e.g., revoked tenants, decommissioned hosts whose IPs get reused) may need shorter bounds — handle that with mTLS identity checks at the destination rather than by failing closed.

4. Failure Modes & Operational Reality#

4.1 Deploy-Time Errors — Routing to Terminated Instances#

The most frequent discovery failure is not an outage. It's a 0.5–5% error spike on every deploy, which teams learn to ignore.

Setup: payments, 400 pods, rolling deploy 25% at a time, callers poll registry every 30s.
       Pod killed on SIGTERM immediately; deregistration via heartbeat expiry (30s).

t=0:      100 pods receive SIGTERM and exit within 1s
t=+1s:    Registry still lists them; callers' caches list them
t=+1–30s: ~25% of new connections to payments are refused; in-flight requests are cut
t=+30s:   Heartbeats expire; registry removes 100 pods
t=+30–60s: Callers poll and converge
t=+60s:   Next batch of 100 pods... repeat 4×
Result:   ~4 minutes of 5–25% errors on callers without retries; retry load on the rest

Detection: upstream_cx_connect_fail, upstream_rq_reset by destination correlated with deploy events; deploy.error_rate_delta.

Mitigation: Pause deploy; enable caller retries on connect-failure (safe — the request never reached the server).

Prevention: Shutdown sequence:

Owner: Platform owns the preStop defaults and convergence SLO; service teams own graceful drain in their server code.

4.2 Mass Deregistration — The Health Check That Emptied a Service#

t=0:      Team adds a readiness check that verifies connectivity to the shared config service
t=+2min:  Deploy reaches 100%; config service has a 20-second GC pause
t=+2min:  All 600 pods fail readiness within 10s; endpoints → 0
t=+2min:  Control plane dutifully pushes 'payments has no endpoints' to 8,000 proxies
t=+2m10s: Every caller gets 'no healthy upstream'; checkout at 100% errors
t=+2m20s: Config service recovers; readiness passes after failureThreshold/successThreshold windows
t=+3min:  Endpoints repopulate; convergence completes
A 20-second blip in a non-critical dependency caused a full payments outage.

Detection: endpoints_ready{service} dropping > 30% in < 60s; no_healthy_upstream_total.

Mitigation: Panic mode in proxies (route to all hosts when < 50% healthy) would have kept serving through the blip. Control-plane guard would have held the update.

Prevention: Readiness checks only local state; dependency health belongs in circuit breakers (Circuit Breakers), not readiness. Health-check changes roll out with canary like code.

Owner: Service team (readiness semantics); platform (panic threshold, mass-removal guard).

4.3 Registry Quorum Loss#

t=0:      2 of 3 etcd members lose disks in a maintenance event; quorum lost
t=+0s:    Writes fail; the orchestrator can't update endpoints; watches stall
t=+0s:    Data plane: every proxy continues with last-known-good — traffic unaffected
t=+5min:  Autoscaler adds 200 pods for rising traffic — they're not routable
t=+10min: 30 pods die from a node failure — outlier ejection removes them per caller
t=+45min: etcd restored from snapshot; controllers reconcile; watches resume
t=+46min: 8,000 proxies reconnect at once → discovery servers CPU-bound

Detection: etcd_server_has_leader == 0, xds_push_age_seconds rising on every proxy, registry_write_errors_total.

Mitigation: Freeze deploys during the outage; scale by adding capacity to existing pods if possible; after recovery, stagger reconnects.

Prevention: 5-member etcd for large clusters, spread across failure domains; tested restore runbook (< 30 min); jittered proxy reconnect; discovery servers that serve from their own cache while the store is down.

Owner: Platform on-call; incident commander freezes deploys.

4.4 Reconnect Storm After Control-Plane Recovery#

Detection: xds_connected_clients sawtooth; discovery_server_cpu pegged; xds_push_errors.

Prevention: Reconnect with exponential backoff and full jitter; resume by version so most proxies need only a delta; discovery servers build one snapshot and serve it to many clients rather than querying the store per client; admission control on full-state requests.

Owner: Platform.

4.5 Stale DNS — The Endpoint That Wouldn't Go Away#

A legacy JVM service with a security manager caches DNS forever. A database proxy's IP changes during a migration; the service keeps connecting to the old IP for 11 days until its next restart — the old host was reused by another service, which now receives (and rejects) the traffic.

Detection: connections_to_decommissioned_ips (from flow logs); destination-side mTLS identity mismatches.

Prevention: Explicit networkaddress.cache.ttl in base images; mTLS so a reused IP can't impersonate a service; decommissioned IPs quarantined for 24h before reuse.

Owner: Platform (base images, IP reuse policy); service team (legacy upgrade).

4.6 Zombie Instance / IP Reuse#

A node partition leaves a pod running but disconnected from the orchestrator. The orchestrator reschedules the pod elsewhere and the old IP is reassigned to a new pod of a different service. Callers with stale views send payments traffic to an inventory pod.

Prevention: mTLS with workload identity (SPIFFE-style) so the caller verifies it reached payments, not whoever holds the IP; endpoint metadata includes a pod UID; servers reject requests addressed to a different service name.

Owner: Platform (identity), security.

4.7 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Knowing Raft in detail ≠ designing discovery. Consensus is one box; propagation and health are the problem.
• "We'll use Istio" ≠ a design. The mesh doesn't decide readiness semantics or panic thresholds for you, and it has its own fan-out limits.
• Comparing Consul/etcd/ZooKeeper feature matrices ≠ judgment. State the CP/AP posture and move on.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Consensus internals (Raft log replication) unless asked
• Registry storage schema beyond an entity table
• Detailed DNS protocol (SRV record fields)
• Vendor comparisons — one line each

6.4 Follow-Up Questions to Expect#

1. "How do you choose the drain wait?" — Convergence p99 plus margin; measured, not guessed.
2. "What happens if the discovery server pushes an empty endpoint list?" — Validation NACKs it; guard holds large removals; panic threshold routes to all.
3. "How does the caller pick among 2,000 endpoints?" — Subsetting (each caller gets ~50–100 deterministic endpoints) plus least-request/P2C; see Load Balancer.
4. "How do you handle a service that's in both old and new registries during migration?" — Dual-register via the platform, callers read the new system first with fallback to the old, migrate by caller cohort.
5. "Why not just use a load balancer?" — At small scale, do. At thousands of services, per-service LBs are costly and add a hop; client-side LB with pushed endpoints scales better.
6. "What's the difference between liveness and readiness?" — Liveness restarts; readiness routes. Never make liveness depend on dependencies — that's a restart storm.
7. "How do you know convergence is healthy?" — Synthetic changes (canary endpoint flipped every minute) and measure time to observe in sampled proxies.

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design service discovery for our microservices platform."

Drill 2: Registry Down#

Prompt: "Your registry cluster has lost quorum. What happens to traffic?"

Drill 3: Make It Concrete — Zero-Error Deploys#

Prompt: "Every deploy of payments causes a 2% error spike for 3 minutes. Fix it."

Drill 4: The Health Check That Emptied a Service#

Prompt: "A readiness check change caused all instances of a service to go unhealthy at once. How do you prevent this class of problem?"

Drill 5: Control-Plane Fan-Out#

Prompt: "Our mesh control plane falls over during large deploys. We have 12,000 sidecars and 2,000 services."

Drill 6: Hot Service With 20,000 Instances#

Prompt: "One service has 20,000 instances and 3,000 callers. Every caller connecting to every instance means 60M connections. What do you do?"

Drill 7: Build vs Buy#

Prompt: "Should we build our own discovery system?"

Drill 8: Changing Discovery Without an Outage#

Prompt: "We need to migrate 800 services from Eureka to Kubernetes-native discovery with a mesh. How?"

Drill 9: Cost#

Prompt: "The mesh sidecars cost us a lot of CPU. Is service discovery worth it?"

Drill 10: Multi-Region#

Prompt: "We're adding a second region. How does discovery work across regions?"

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Autoscaling Doesn't Help#

Context: During a product launch, traffic to the search service triples. The autoscaler adds 300 pods within 4 minutes, but search latency keeps climbing and callers see timeouts for 12 minutes. The new pods show near-zero traffic. The on-call escalates to you.

Questions to Surface First:

• Are the new pods passing readiness? How long do they take to become ready?
• What's the control plane's push latency right now — is it backlogged by the scale-up itself?
• Are callers subset-based? Do new pods appear in any caller's subset?
• Are callers holding long-lived connections (HTTP/2, gRPC) that never rebalance to new pods?

Typical L5 Approach: Checks that pods are running and ready, restarts a few callers to see if traffic shifts, adds more pods. Restarts help anecdotally, which hides the real cause.

Staff Approach: Measures each pipeline stage: pod ready time (warm-up took 90s due to a large index load), control-plane push latency (backlogged to 45s by 300 simultaneous changes pushed as full-state updates), and caller connection behavior (gRPC channels with long-lived HTTP/2 connections never opened connections to new endpoints because the client's LB policy was pick-first). Fixes the LB policy to round-robin/least-request over all endpoints, enables incremental pushes with debounce, and moves index loading before readiness.

Principal Approach: Recognizes that "autoscaling works" was never tested end-to-end: time from scale decision to useful traffic on new capacity. Defines a platform SLO — capacity-to-traffic time p99 < 60s — and a quarterly test that scales a tier-1 service 2× under synthetic load and measures it. Makes the gRPC LB policy a platform default rather than a per-service choice.

Key Takeaway: "Discovery isn't done when the registry knows about new instances. It's done when traffic reaches them."

What clears the Staff bar:

• Measures each stage of scale-to-traffic separately
• Finds the connection-level cause (sticky HTTP/2 connections)
• Converts the fix into a platform default

Deep Dive 2: Silent Staleness#

Context: A security review discovers that for 6 weeks, ~3% of sidecars in one cluster have been running with endpoint data 2–9 days old. They were routing mostly correctly because the underlying services were stable. No alert fired. How did this happen and what do you change?

Questions to Surface First:

• Why did these sidecars stop receiving updates — a crashed watch, a NACKed config, a bug in resume logic?
• What does the data plane do when it NACKs a push — keep the previous config forever?
• Is there any metric for config age per proxy? Is anyone looking at it?
• Were any requests misrouted — e.g., to reused IPs?

Typical L5 Approach: Finds the bug (a specific config in one namespace was rejected by an older sidecar version, which NACKed every subsequent push containing it), upgrades the sidecars, closes the ticket.

Staff Approach: Fixes the bug, then the detection gap: every proxy reports config_version and last_update_age_seconds; alert when > 1% of proxies are more than 5 minutes behind the latest version, or any single proxy is > 1 hour behind. Makes NACK a first-class signal with a per-resource error, so one bad resource doesn't block every update. Adds a synthetic "canary endpoint" flipped every minute to measure end-to-end convergence continuously.

Principal Approach: Treats fail-static's hidden cost explicitly: the property that saves you during outages also hides failures during normal operation. Every fail-static system in the org (discovery, config, certificates, feature flags) must report staleness, and the platform's quarterly review includes a staleness distribution. Sets a policy for version skew between control plane and data plane (e.g., N−2 max).

Key Takeaway: "Fail static is a feature during outages and a bug when nobody measures staleness."

What clears the Staff bar:

• Connects the resilience property to its failure mode
• Adds per-proxy staleness telemetry and a continuous convergence probe
• Handles NACK granularity

Deep Dive 3: Onboarding a Large Service Into the Mesh#

Context: The ads-serving team (8,000 pods, 400K RPS, p99 budget of 15ms) wants to join the mesh for mTLS and uniform discovery. They're worried about the sidecar's latency and the control-plane load their churn will add — they autoscale by ±30% daily.

Questions to Surface First:

• What's their current discovery mechanism and latency overhead?
• What's the sidecar's measured p99 overhead at their request size and QPS?
• How many callers do they have, and how many services do they call?
• Can they use proxyless xDS instead of a sidecar?

Typical L5 Approach: Deploys sidecars with default resources, measures, finds +1.5ms p99, and the ads team rejects the mesh.

Staff Approach: Proposes proxyless gRPC xDS for ads (same control plane, no extra hop), with mTLS via the xDS-provided certificates. Handles their churn with subsetting (each caller sees ~100 of 8,000 pods) and debounced incremental pushes. Validates in shadow: control-plane load test simulating their daily ±30% swing, before onboarding.

Principal Approach: Recognizes this as a platform product question: the mesh must offer tiers (sidecar, proxyless, per-node proxy) with clear latency and feature tradeoffs, or the highest-value services will opt out and the platform fragments. Owns the published tier matrix and the latency SLO for each.

Key Takeaway: "A platform that only has one shape loses its most important customers."

What clears the Staff bar:

• Offers proxyless as an alternative rather than forcing the default
• Quantifies churn impact on the control plane before onboarding
• Uses shadow mode and cohort rollout

Deep Dive 4: Post-Mortem — Multi-Day Outage in the Discovery Layer#

Context: Your company's discovery and configuration platform — a Consul-like system that almost every service depends on for both endpoints and key/value config — degraded after enabling a new streaming feature under record load. The store's performance collapsed, and because services required it at startup and for config reads, recovery took days. (The public Roblox post-mortem from October 2021 describes a 73-hour outage with a similar shape: a Consul streaming change plus a storage-layer pathology under high load.) You're leading the post-mortem.

Questions to Surface First:

• Which services required the discovery system at startup versus only for updates?
• Was the same cluster used for discovery, config, and leader election — one failure domain for three functions?
• Why couldn't the team roll back quickly — was the feature change entangled with data format changes?
• Did monitoring depend on the system that failed?

Typical L5 Approach: Fixes the specific bug (disable the new feature, tune the storage layer), adds more capacity to the cluster, adds alerts on its latency.

Staff Approach: Separates functions and removes startup dependencies. Endpoint discovery, dynamic config, and coordination move to separate clusters with separate failure postures. Services boot with a cached/baked config and last-known endpoints, so a discovery outage doesn't prevent restarts. Monitoring and incident tooling get an independent path. Feature changes to the platform roll out per-cluster with dwell time and a tested rollback.

Principal Approach: Frames the discovery/config platform as the company's single largest correlated-failure risk. Requires cell architecture — multiple independent clusters, each serving a subset of services — so no single platform failure takes out more than a fraction of the business. Sets a "cold start without control plane" requirement for tier-1 services and tests it quarterly. Budgets the headcount for it explicitly, because this is an insurance purchase that leadership must approve.

Key Takeaway: "The worst discovery outages aren't about routing. They're about everything else that quietly started depending on the discovery cluster."

What clears the Staff bar:

• Separates functions by failure posture
• Removes startup dependencies
• Makes monitoring independent of the failed system

Deep Dive 5: Multi-Region Expansion#

Context: The company is adding two regions to its one. Leadership wants "any service can call any service in any region." The platform team proposes one global discovery cluster stretched across all three regions.

Questions to Surface First:

• What's inter-region RTT (typically 30–150ms)? A stretched consensus cluster pays that on every write.
• What happens to each region when the WAN partitions?
• Do services actually need cross-region calls, or is it "just in case"?
• Which services hold regional data, and would a cross-region call even return the right answer?

Typical L5 Approach: Accepts a stretched cluster with 5 nodes across 3 regions for global consistency.

Staff Approach: Rejects the stretched cluster: writes pay WAN RTT on every endpoint change, and a partition can leave a region without quorum — freezing its discovery for reasons unrelated to its own health. Proposes one registry per cluster (authoritative locally), federated read-only views of other regions, locality-first routing, and cross-region calls only for an allowlist with capacity caps.

Principal Approach: Pushes back on the requirement: "any service can call any region" becomes an implicit global dependency graph that makes every region's availability depend on every other's. Proposes regional independence as the default contract, with explicit, reviewed cross-region dependencies. Prices the design options — per-region stacks cost roughly N× platform infrastructure, but a stretched design risks correlated global failure.

Key Takeaway: "Consensus across regions makes every region's discovery depend on the WAN. Keep authority local; share views, not quorums."

What clears the Staff bar:

• Rejects stretched consensus with a concrete reason (WAN RTT, partition quorum)
• Keeps authority local and federation read-only
• Caps cross-region failover by capacity

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Explain why the data plane must fail static and never read the registry per request
• Distinguish endpoint discovery (AP-friendly) from coordination discovery (CP-required) and design each
• Derive a deploy drain wait from convergence p99 and describe the full shutdown sequence
• Explain layered health (readiness, outlier ejection, panic threshold) and why readiness must not check shared dependencies
• Compute control-plane fan-out and fix it with scoped, incremental, debounced pushes
• Explain subsetting for very large services
• Design multi-cluster discovery with local authority, federation and capacity-capped failover
• Split ownership: platform registers and distributes; service teams own readiness and graceful shutdown

The Bar for This Question#

Mid-level (L4): Knows registries exist; describes register-heartbeat-lookup. Might suggest DNS.

Senior (L5): Picks a reasonable registry, explains heartbeats and client caching, mentions health checks and load balancing. Misses fail-static as a principle, deploy-time errors, mass deregistration, control-plane fan-out, and ownership.

Staff+ (L6): Separates control and data planes, designs for churn with a convergence budget, layers health with a panic threshold, handles control-plane failure and recovery storms, scopes fan-out, keeps authority per cluster, and splits ownership between platform and service teams. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Service Discovery Should Be AP — Even If the Registry Is CP"#

The Staff position: Let the registry be CP if it's convenient; make the system AP by caching everything in the data plane.

Why this matters in interviews: It resolves the classic "Eureka vs ZooKeeper" debate by separating the planes — and shows you know the debate exists.

10.2 "Self-Registration Is an Anti-Pattern on Modern Platforms"#

Every language needs a correct implementation of registration, heartbeating and graceful deregistration. The orchestrator already knows what's running and whether it passed readiness. Third-party registration from the orchestrator has one implementation, one owner, and one set of bugs.

The Staff position: Platform registers; services expose readiness. Self-registration only where no orchestrator exists.

Why this matters in interviews: It's a clean ownership argument that interviewers rarely hear.

10.3 "DNS Is a Terrible Service Discovery System That You Must Support Forever"#

The Staff position: DNS is the compatibility layer for everything that can't speak your control-plane protocol — databases' clients, third-party tools, legacy code. Support it; don't design around it.

Why this matters in interviews: Saying "DNS with a short TTL" as the whole answer is a common L5 signal; saying why it's only a fallback is L6.

10.4 "Health Checks Cause More Outages Than They Prevent"#

Readiness checks that probe shared dependencies, liveness checks that restart healthy-but-busy processes, and fleet-wide health config changes have caused more full-service outages than the dead instances health checks remove — which outlier detection would have caught anyway.

The Staff position: Health checks should be local, conservative, and capped (panic thresholds, removal guards). Caller-observed failure is the more reliable signal.

Why this matters in interviews: It shows you've debugged a mass deregistration — or read enough post-mortems to know the pattern.

10.5 "Your Discovery Cluster Is Secretly Your Config System, Lock Service, and Monitoring Dependency"#

Convenience accretes: "it's already there, let's store feature flags in it", "let's use it for leader election", "let's have the metrics agent discover targets from it". Each is reasonable; together they make one cluster a single point of failure for functions with different availability needs.

The Staff position: One cluster per failure posture. Audit what depends on the discovery cluster every quarter.

Why this matters in interviews: It's the Principal-flavored insight you can deliver at Staff level.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

At L6, discovery is a well-designed system with a fail-static data plane. At L7, discovery is the nervous system of the company's infrastructure — the one control plane every service touches, the place where security identity, traffic policy, config and dependency graphs converge, and therefore the org's largest shared failure domain. The Principal questions are about scope and time: how many discovery systems should exist (usually one, sometimes one per failure posture), what it should refuse to become (a config store, a lock service), which of its contracts are forever (service naming, identity format, the manifest), and how to make it survivable when — not if — it fails.

The Org-Level Fault Line#

One discovery platform vs many.

🧭 Principal Move: "I want exactly one way to name, identify and discover a service — and more than one cluster implementing it. Standardize the contract; distribute the failure domain."

Cost Model#

Assumptions: control-plane nodes ~8 vCPU each; ~$30/vCPU-month; sidecar ~0.15 vCPU + 75MB per pod where used; fully loaded engineer ~$300K/year. Order of magnitude only.

The line item that surprises leadership is the data plane, not the registry. The strategic decision is sidecar vs proxyless vs per-node proxies, which can swing data-plane cost by 3–5×.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Service Discovery & Routing Standard v1

Scope: All production services and clients in all clusters.

Requirements:

• Services MUST be registered by the platform from orchestrator state; self-registration is prohibited except on approved non-orchestrated hosts.
• Services MUST expose a readiness endpoint that checks only local state. Readiness MUST NOT depend on shared downstream services.
• Services MUST implement graceful shutdown: fail readiness on SIGTERM, continue serving for ≥ the published convergence p99 + 2s, then drain.
• Data-plane components MUST route from local state and MUST continue with last-known-good state if the control plane is unreachable. Last-known-good MUST survive process restart.
• The control plane MUST NOT remove more than 30% of a service's endpoints within 60s without human confirmation; proxies MUST use a panic threshold of 50%.
• Services SHOULD declare their dependencies in the service manifest; proxies subscribe only to declared dependencies.
• Discovery clusters MUST NOT be used for application config, locks or leader election.

Exceptions: Filed with the platform team; reviewed monthly; 6-month expiry.

Success metrics: Convergence p99 < 5s during peak deploys; deploy-attributable error rate < 0.01%; % of proxies with config age > 5 min < 0.1%; zero incidents where control-plane failure stopped existing traffic.

What I'd Tell the VP#

"Every service we run finds every other service through one system, which makes it both essential and our biggest single risk. We've designed it so that if it fails, the traffic that's already flowing keeps flowing — we lose the ability to deploy for a while, not the business. Over the next year we'll retire our three older discovery systems into one, and split it into independent cells so a failure affects a fraction of services, not all of them. The main cost is the proxy fleet, and we have a plan to cut that by moving our heaviest services to a lighter-weight option. The outcome you'll see is deploys that don't cause customer errors and no company-wide outages from this layer."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• An excellent single-cluster design with no answer for how 20 clusters and 3 regions share naming and identity.
• "We'll migrate everything to the mesh" without a plan for the long tail or the cost of running two systems meanwhile.
• Treating the discovery cluster's other uses (config, locks) as someone else's problem.

Appendices

Appendix A: Mechanics in Depth#

A.1 Registration Patterns#

A.2 Failure Detection Math#

Heartbeat-based removal time ≈ heartbeat_interval × missed_threshold + propagation
  Eureka-style: 30s × 3 + client refresh 30s ≈ up to 120s
Readiness-based removal (tuned): probe_period 2s × failureThreshold 2 + push 2s ≈ 6s
Outlier ejection (caller-local): 5 consecutive 5xx at 500 RPS ≈ 10ms; at 1 RPS ≈ 5s
Graceful shutdown: 0s detection (the pod fails its own readiness) + push p99

Caller-side outlier detection is the fastest signal for high-traffic callers and the slowest for low-traffic ones; the registry is the opposite. Use both.

A.3 Incremental Push With Versions#

Subscriber state: {service → version}
On connect:  Watch(services, known_versions)
Server:      for each service: if known_version == current: nothing
                               elif delta_log has known_version: send delta
                               else: send full state
On change:   debounce 100ms–1s per service, then fan out delta to subscribers of that service
Client:      apply; ACK(version) or NACK(version, error) and keep previous

A.4 Deterministic Subsetting#

def subset(backends, client_id, subset_size):
    subset_count = ceil(len(backends) / subset_size)
    round = client_id // subset_count
    shuffled = shuffle(sorted(backends), seed=round)     # same shuffle for a round of clients
    subset_id = client_id % subset_count
    return shuffled[subset_id * subset_size : (subset_id + 1) * subset_size]

Each "round" of clients covers every backend exactly once, so load is balanced across backends while each client holds only subset_size connections.

Appendix B: Naming, Identity and Data Model#

Rule: Route by name, verify by identity, never trust an IP.

Appendix C: Coordination Mechanisms — Registry Options#

C.1 Quick Comparison#

See ZooKeeper & etcd and Distributed Consensus.

Appendix D: Client Behavior Contract#

D.1 Client/Proxy Requirements#

D.2 Reconnect Storm Prevention#

backoff = min(60s, 1s × 2^attempt) × random(0.5, 1.5)
on reconnect: send known_versions → receive deltas, not full state
server: admission control — at most N concurrent full-state syncs per replica

D.3 Graceful Shutdown Contract#

SIGTERM →
  set readiness = false
  sleep(convergence_p99 + 2s)        # e.g., 7s
  stop accepting new connections; send GOAWAY / Connection: close
  wait for in-flight (max 15s)
  exit
terminationGracePeriodSeconds ≥ sleep + drain + margin (e.g., 30s)

Appendix E: Observability#

E.1 Core Metrics — Non-Negotiable#

# Control plane
registry_write_latency_p99, registry_has_leader
xds_push_latency_seconds{p50,p99}
xds_connected_clients, xds_nack_total{resource}
endpoint_changes_per_minute{service}
mass_removal_guard_triggered_total{service}

# Data plane
proxy_config_age_seconds (distribution across proxies)
endpoints_healthy_ratio{service}  (per caller)
panic_mode_active{service}
outlier_ejections_active{service}
upstream_cx_connect_fail{service}, no_healthy_upstream_total{service}

# End-to-end
canary_endpoint_convergence_seconds       # synthetic change, observed in sampled proxies
deploy_attributable_error_rate{service}

E.2 Critical Alerts#

E.3 Control Plane vs Data Plane#

Control-plane dashboards tell you whether changes are flowing; data-plane dashboards tell you whether traffic is flowing. During a control-plane outage, the first set is red and the second should be green. If both go red together, the fail-static property is broken — that is a Sev-1 design bug, not just an outage.

E.4 Debugging "Traffic Isn't Reaching New Instances"#

1. Is the pod ready? (pod_ready_seconds)
2. Is it in the registry? (endpoint slice contents)
3. Did the push reach callers? (proxy_config_age, sampled endpoint tables)
4. Are callers opening connections to it? (long-lived HTTP/2 connections, pick-first LB, subsetting)
5. Is it in any caller's subset?

Appendix F: Scale Evolution#

F.1 What Works at Each Scale#

F.2 Multi-Region Path#

Per-cluster authority → read-only federation → locality-aware routing with capped remote failover → edge-level regional evacuation. Never stretched consensus.

F.3 What You Don't Build on Day One#

• Your own registry or xDS server
• Cross-region federation before a second region exists
• Subsetting before any service exceeds ~1,000 instances
• Proxyless xDS before the sidecar tax is measured
• A custom service manifest format before you have 50+ services

Appendix G: Multi-Tenancy, Fairness and Cost#

G.1 Control-Plane Fairness#

One namespace's churn (a team autoscaling every 30 seconds) can dominate control-plane capacity. Rate-limit endpoint changes per namespace (e.g., ≤ 200/min before debounce extends), and prioritize tier-1 services' pushes.

G.2 Registry Write Quotas#

Per-namespace quotas on registered objects and write rate protect the consensus store (etcd performs best well under its 2–8GB size range).

G.3 Cost Levers#

G.4 Tradeoff Summary#