Technologies referenced in this case study: API Gateways · ZooKeeper & etcd · Redis

Related: API Gateway · Service Discovery · CDN & Edge Caching · Circuit Breakers · Rate Limiting · Multi-Region · Real-time WebSockets · Degraded Mode · Consistent Hashing · Networking · Numbers to Know

How to Use This Case Study#

This case study is organized for the interview first and for reference second. Read it front-to-back once; then return to the fault lines and deep dives that match your weak spots. If you need the packet-level background — TCP, TLS, HTTP/2, anycast — start with Networking.

Without a designed balancing and health policy:
t=0:        Backend b-17 loses its database connection pool. Every request now
            returns 503 in 2 ms instead of 200 in 40 ms.
t=+1s:      Least-connections LB sees b-17 always has 0 active requests.
            Routes more and more traffic to it.
t=+10s:     b-17 receives ~35% of all requests across a 40-host pool. 35% error rate.
t=+30s:     Active health check (GET /health every 10s, needs 3 failures) still
            passes: /health doesn't touch the database.
t=+8min:    Someone notices the error graph, finds b-17, terminates it by hand.

With passive outlier detection and P2C least-request:
t=0:        b-17 starts returning 503 in 2 ms.
t=+0.5s:    5 consecutive 5xx → b-17 ejected for 30s (cap: 10% of pool ejectable).
t=+0.5s:    Error rate peak: ~0.6% for half a second. Traffic spread over 39 hosts.
t=+30s:     b-17 re-admitted on probation, fails again, ejected for 60s.
t=+2min:    Alert: host b-17 ejected 3 times → owner's auto-remediation replaces it.

Executive Summary

If you only read one section, read this. Everything in this case study flows from the contrast below.

What This Interview Actually Tests#

A load balancer is not an algorithm question. Round-robin works fine on most days.

It is a failure-judgment question: a load balancer decides, many times per second and without a human, which machines deserve traffic — and the most common way it fails is by being confidently wrong at scale. It tests:

• Whether you know what each layer (DNS, anycast, L4, L7, client-side) can and can't see, and put each decision at the layer that has the information
• Whether you know where the latency actually goes — the proxy hop is sub-millisecond; the TLS handshake and connection setup are not
• Whether your health model distinguishes "dead" from "slow" from "fast-failing", and refuses to eject half the fleet during overload
• Whether you treat the load balancer's own config and control plane as the largest blast radius in the system
• Whether connection lifetime — keepalive, HTTP/2, WebSockets — is part of your balancing model

The key insight: The load balancer sits on every request, so its mistakes are correlated by construction. A backend bug hurts one host's share of traffic; a load-balancer bug — a bad health check, a herding algorithm, a config push — hurts all of it at once. Staff candidates design the LB to fail less confidently: bounded ejection, panic thresholds, staged config rollout, and draining that respects connection lifetime.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

Three intents produce three different load balancers. Name them, then commit.

🎯 Staff Move: "I'll design the internet edge for a latency-sensitive HTTP API — that's where TLS cost, L4/L7 layering and health judgment all matter. I'll sketch the L4 tier but not redesign packet forwarding, and I'll note where east-west balancing differs: it moves into the client, because a central hop for every internal call is a tax we don't need to pay."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Naming how real edge networks settled these fault lines shows you've studied operations, not just cloud console options.

Google — Maglev#

Google's Maglev paper (NSDI 2016) describes a software L4 load balancer running on commodity servers. Routers spread packets across Maglev machines with ECMP; each machine uses Maglev hashing — a consistent hashing scheme built on a fixed-size lookup table populated by per-backend permutations — plus a local connection-tracking table, so packets of a flow reach the same backend even when the Maglev fleet or backend set changes. The fleet is active-active, scaling out instead of using active-passive pairs.

Staff insight: Maglev's design point is minimal disruption under change: when backends or balancers come and go, almost all existing flows keep their mapping. In an interview, "consistent hashing plus connection tracking at L4" answers the question "what happens to in-flight connections when you add a load balancer?"

Meta — Katran#

Meta open-sourced Katran in 2018, an L4 load balancer built on XDP and eBPF that processes packets in the kernel's earliest receive path. It sits behind ECMP, uses a Maglev-style consistent hash with a connection table, and encapsulates packets to backends, which reply directly to clients (direct server return) so return traffic bypasses the balancer.

Staff insight: Direct server return matters because responses are usually much larger than requests. If return traffic doesn't flow through the L4 tier, that tier is sized for inbound packets only — often a 5–10× smaller fleet. Mention it when sizing the L4 tier.

Lyft — Envoy#

Envoy was built at Lyft and open-sourced in 2016 as an L7 proxy used both at the edge and as a sidecar next to every service. Its load balancer offers round robin, least-request using power-of-two-choices by default, ring hash and Maglev for affinity, passive outlier detection, a panic threshold (by default, if fewer than 50% of hosts are healthy, it balances across all hosts), and dynamic configuration over the xDS APIs from a control plane.

Staff insight: Envoy's defaults encode the hard-won lessons in this case study — P2C over least-connections, outlier detection with a capped ejection percentage, panic mode. Citing why those defaults exist is far stronger than citing that the product exists.

What Interviewers Probe#

System Architecture Overview#

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The walkthrough below is a 45-minute script. The words in italics are meant to be said out loud.

Phase 1: Requirements & Framing (2–3 minutes)#

"Load balancer can mean three different things: the internet edge in front of a public service, balancing between internal services, or a pure L4 packet balancer. I'll design the internet edge for a latency-sensitive HTTP API, where TLS, L4/L7 layering and health judgment all matter. I'll mention how east-west differs at the end."

"99.99% gives the edge about four minutes a month. A single bad global config push can eat that in one go, so I'll treat config safety as a first-class part of the design, not an operations footnote."

Phase 2: Core Entities & API (1–2 minutes)#

# Control-plane API (service teams own routes; platform owns listeners and templates)
PUT  /v1/routes/{service}            { host, path_prefix, cluster, timeout_ms, retry: {...} }
PUT  /v1/clusters/{service}          { lb_policy: "p2c_least_request" | "ring_hash", hash_on?, ... }
POST /v1/endpoints/{service}/drain   { endpoint, drain_seconds }
GET  /v1/config/versions/{v}/status  → { stage: "canary" | "zone" | "region" | "global", health_delta }
POST /v1/config/rollback             { to_version }        # platform on-call only

"Routes are owned by service teams; listeners, TLS policy and the defaults template are owned by the edge platform. That split matters later: it's what lets a team ship a route change without being able to change the timeout behavior for everyone else."

Phase 3: High-Level Architecture (≤5 minutes)#

"GeoDNS or anycast picks a region. In the region, routers ECMP across an L4 tier that uses Maglev-style consistent hashing with connection tracking, so flows stay stable when L4 nodes or L7 proxies change. The L7 fleet is stateless, terminates TLS, routes by host and path, and balances to backends with P2C least-request and outlier detection. The control plane pushes endpoints and config incrementally. I'll spend the depth on three places: where the latency goes, how health decisions fail, and how config changes fail."

Phase 4: Transition to Depth (1 minute)#

"The boxes are well known. Where this design succeeds or fails is the TLS cost at the edge, the health and balancing policy toward backends — because that's where load balancers amplify failures — and the config pipeline, because it's the one change that touches every request. Which would you like first?"

Phase 5: Deep Dives (25–30 minutes)#

Deep dive 1 — Where the latency goes (7 min).

"For a new client 80 ms away, two round trips — TCP and TLS 1.3 — cost 160 ms before the first byte of the request arrives. The proxy's own processing is well under a millisecond. So the levers are: terminate TLS as close to the user as possible, maximize connection reuse with HTTP/2 and long keepalives, and support session resumption so returning clients skip the certificate exchange. On the server side, a full handshake with RSA-2048 costs roughly ten times the CPU of ECDSA P-256; serving ECDSA certs to clients that support them is the biggest single lever on edge fleet size."

"Toward backends, the proxy keeps warm connection pools, so the backend hop is ~0.1–0.5 ms in-AZ with no handshake. If the backends are in another AZ, add ~1–2 ms round-trip — which is why I'd keep zone-aware routing on and only spill across zones when local capacity is short."

Deep dive 2 — Balancing and health (12 min).

"Balancing is P2C least-request: pick two healthy hosts at random, send to the one with fewer in-flight requests. With 50 proxies each choosing independently, randomness prevents them from herding onto the same host. New hosts get a slow-start weight that ramps over 60 seconds, so they warm caches and JIT before taking a full share."

"Health has two signals. Active checks every 5 seconds hit a shallow /healthz — the process is up and accepting — and decide admission. Passive outlier detection watches real responses: five consecutive 5xx or a success rate more than ~2 standard deviations below the pool ejects the host for 30 seconds, doubling on repeat. Two guards stop the LB from making things worse: no more than 10% of a pool can be ejected by outlier detection, and if fewer than 50% of hosts are healthy, we enter panic mode and spread across all of them. When half the fleet looks sick at once, the signal is wrong far more often than the fleet is."

Deep dive 3 — Config safety (6 min).

"Config changes — routes, clusters, TLS policies, certs — flow through one pipeline: schema validation, semantic checks (does every route point to an existing cluster? does any timeout exceed the listener's?), then a staged rollout: one canary proxy per region, then one AZ, then one region, then global, with a bake time of 5–10 minutes and automatic rollback if lb.upstream_5xx_rate or lb.added_latency_p99 worsens beyond a threshold. Proxies keep the last-known-good config and refuse a config that fails to load, rather than crash. The data plane must keep serving if the control plane is down."

Deep dive 4 — Draining and long-lived connections (3 min).

"On deploy, a backend is marked draining: no new requests, in-flight finish, HTTP/2 connections get GOAWAY so clients move. For short APIs the drain window is ~2× p99 request time — 10–30 seconds, not the 300-second default many cloud balancers ship with. WebSockets are different: they need an application-level 'reconnect' message and jittered reconnection across 5–10 minutes, or the drain becomes a reconnect storm."

Phase 6: Wrap-Up (2–3 minutes)#

"Summary: two tiers — L4 with consistent hashing for stable flows, L7 for TLS and per-request decisions. P2C least-request with outlier detection, capped ejection and panic mode, so the balancer can't amplify a bad signal into an outage. TLS cost managed with ECDSA and resumption. Config shipped like code with staged rollout and automatic rollback. Next I'd build: east-west balancing in a sidecar with the same policy template, and global steering that shifts load between regions on capacity, not just health."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Every candidate has used a load balancer; very few have been paged because one made an incident worse. Interviewers use this question to separate candidates who think of the LB as plumbing from those who think of it as an automated decision-maker with fleet-wide authority. The Staff-level signal is recognizing the specific ways that authority fails: herding, over-ejection, sinkholes, correlated health checks, reconnect storms, and config pushes.

It's also a layering question. DNS, anycast, L4, L7 and client-side balancing each see different information and fail differently. Putting a decision at the wrong layer — trying to do request-level health at DNS, or connection-level stickiness at L7 for UDP — is a design error that no amount of tuning fixes.

1.2 The L5 vs L6 Contrast — Visual#

The Senior path builds a balancer that works when its inputs are true. The Staff path builds one that stays safe when its inputs are wrong.

1.3 The Staff Question That Cuts Through Everything#

"When this load balancer is wrong, how many requests does it get wrong — and how quickly does it stop?"

That one question surfaces the herd effect (wrong about which host is least loaded → all proxies pick it), over-ejection (wrong about health → removes capacity during overload), config pushes (wrong about routes → every request), and draining (wrong about when a host is gone → resets). Each part of the design either bounds the blast radius of a mistake or shortens its duration.

🎯 Staff Move: "I want every automated decision in the balancer to have a bound: ejection capped at 10% of the pool, panic mode below 50% healthy, retries capped at 20% of traffic, and config rolled out one cell at a time. The LB will be wrong sometimes. I'm designing for how wrong and for how long."

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Internet edge (north-south). Clients are many, far away and untrusted. The edge terminates TLS for millions of connections, absorbs abuse, routes by host and path to dozens of services, and is the place where a new connection's latency is mostly network round trips. The design centers on TLS economics, global steering, L4/L7 layering and config safety. This is the intent this case study commits to. For request-level policy on top of the edge — auth, quotas, transformations — see API Gateway.

Service-to-service (east-west). Clients are your own services, a millisecond or less away. Adding a central proxy hop doubles network traversals for every internal call. The design moves balancing into the caller — a library or a sidecar proxy — fed by service discovery. The hard problems shift to control-plane fan-out (pushing endpoint updates to 20,000 sidecars), consistent policy across languages, and retry budgets so a failing service isn't hit by every caller's retries at once.

L4 network load balancing. The unit is the packet or the connection, not the request. Use cases: non-HTTP protocols, UDP, extremely high packet rates, and preserving client IPs. The design centers on forwarding performance (kernel bypass, XDP), consistent hashing with connection tracking so flows survive changes, and direct server return so responses skip the balancer. It's the first tier of the edge design, but as a standalone problem it's a different interview.

2.2 When NOT to Use a (Dedicated) Load Balancer#

🎯 Staff Move: "For east-west traffic I'd remove the load balancer, not scale it. Each internal call through a central proxy pays an extra hop and shares a failure domain with every other service."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: Where to Terminate — L4 vs L7#

L4 is cheap per byte and blind to requests. L7 sees everything and pays for it in CPU, mostly at connection setup. The question isn't which one to use — it's which decisions go where, and who pays for the CPU.

The Staff default: two tiers. L4 with consistent hashing and DSR in front; L7 terminating TLS and re-encrypting to backends where policy requires (mTLS inside the network). Passthrough as a reviewed exception.

When to deviate: a single-service product at < ~20K RPS can use one managed L7 balancer and skip the L4 tier entirely — the cloud provider's L4 is effectively in front already. Non-HTTP protocols (game servers on UDP, MQTT) go L4 only.

🎯 Staff Move: "L4 for things that need to be fast and stable — flows and floods. L7 for things that need to be smart — routing, retries, health from real responses. And I'd re-encrypt to the backend rather than pass TLS through, unless a service has a compliance reason to own its keys."

3.2 Fault Line 2: Even Spread vs Affinity#

The Staff default: P2C least-request everywhere; consistent hashing with bounded load only for routes where affinity measurably pays (a local cache with > 2× hit-rate improvement); no cookie stickiness — move session state to a shared store such as Redis.

When to deviate: a backend tier with large per-user in-memory state (real-time collaboration, game sessions) genuinely needs affinity; route by entity ID with consistent hashing and accept that rebalancing is a migration.

🎯 Staff Move: "Affinity is a cache-hit-rate optimization, so I'd only pay for it where I can measure the hit rate. Everywhere else, P2C — and if someone wants sticky sessions to hold session state, I'd rather fix where the state lives."

3.3 Fault Line 3: Trusting Health Signals vs Ignoring Them#

Health checking is a classifier with two error modes. False negatives keep a broken host in rotation — users see errors. False positives remove a healthy host — the remaining hosts get more load, which can cause more false positives. The second error mode compounds; the first doesn't.

The Staff default: shallow active checks (5 s interval, 2 failures to mark down, 2 successes to mark up) for liveness and admission; passive outlier detection (5 consecutive 5xx or success rate > 2σ below peers) for correctness; ejection capped at 10% of hosts; panic below 50% healthy.

When to deviate: for pools of fewer than ~5 hosts, a 10% cap means zero hosts ejectable — raise the cap to allow one host, and alert on any ejection.

🎯 Staff Move: "An LB that ejects aggressively is great when one host is bad and catastrophic when the signal is bad. I'll cap ejection at 10% and stop trusting health entirely below 50% healthy, because at that point the checks are much more likely to be wrong than half my fleet."

3.4 Fault Line 4: Central Proxy vs Client-Side Balancing#

The Staff default: central L7 for north-south; sidecars or proxyless clients for east-west, all fed by the same control plane and the same policy template.

When to deviate: organizations with < ~30 services often do fine with a client library in one language and DNS-based discovery; a mesh is a platform commitment of several engineers.

🎯 Staff Move: "The edge is a central proxy because clients are untrusted and far away. Inside, I'd push balancing to the caller because it's the only place with per-call latency data and no extra hop — but with one policy template, so a timeout default means the same thing everywhere."

3.5 Fault Line 5: Connection Longevity vs Rebalancing#

Long-lived connections are good for latency (no handshakes) and bad for balance (load is pinned to whoever accepted the connection). HTTP/2 and gRPC multiplex thousands of requests on one connection; WebSockets live for hours.

The Staff default: client→edge connections capped at a max age (~10 min ± jitter) via GOAWAY; edge→backend balancing per request over pooled connections; WebSockets get an application "reconnect" frame and jittered reconnection spread over 5–10 minutes during drains.

When to deviate: for very latency-sensitive mobile clients on poor networks, lengthen the max age (30–60 min) — each reconnect costs them 2+ RTTs at 200 ms each.

🎯 Staff Move: "I'll balance per request, not per connection, wherever HTTP/2 is involved — otherwise I've built a load balancer that balances once per connection lifetime. And I'll cap connection age so a scale-out actually reaches new hosts within minutes."

4. Failure Modes & Operational Reality#

4.1 The Fast-Fail Sinkhole#

A backend breaks in a way that makes it fast. Load-aware balancing rewards speed, so the broken host attracts traffic.

t=0:        Backend b-22 (of 40) loses its connection to the config service after a
            cert expires. Handler returns 503 in 1.5 ms. Healthy p50: 35 ms.
t=+1s:      Least-request: b-22 has ~0 in-flight at all times. Selected whenever it's
            one of the two choices — and wins every comparison.
            Share of traffic: 2.5% → ~5% (P2C limits the damage; least-conn would hit ~30%+).
t=+1s:      Outlier detection: 5 consecutive 5xx on each proxy → b-22 ejected on most proxies.
t=+2s:      Error rate: peak 4.8% for ~1 s, back to 0.05%.
t=+30s:     b-22 re-admitted, fails again, ejected 60 s, then 90 s.
t=+5min:    Alert: lb.host_ejections{host=b-22} > 3 in 5 min. Auto-replace triggers.

Detection: lb.upstream_5xx_rate by host, lb.host_ejections, latency distribution by host (a host that is suddenly much faster than peers is a signal).

Mitigation: outlier detection (consecutive 5xx + success-rate relative to peers); P2C limits a sinkhole's share to roughly 2/N even before ejection, vs a much larger share under least-connections.

Prevention: health endpoints that exercise the critical in-process dependency state (without calling remote shared dependencies); cert-expiry alerts at 30/14/7 days.

Owner: service team owns the bug and its health endpoint; edge platform owns the outlier policy defaults.

4.2 The Health-Check Cascade#

t=0:        Shared auth service p99 rises from 20 ms to 3 s (GC pause storm).
t=+5s:      140 services' /health endpoints call auth to "verify dependencies."
            Health checks time out (2 s timeout). Hosts start failing checks.
t=+15s:     Pools drop below healthy thresholds. LBs without a panic threshold remove
            hosts; remaining hosts take more load, fail checks faster.
t=+30s:     34 services at 0 healthy hosts → LB returns 503 "no healthy upstream"
            for 100% of their traffic. Auth itself has recovered.
t=+90s:     Hosts pass 3 consecutive checks again; pools refill gradually.
t=+4min:    Full recovery. A 30-second auth blip became a 4-minute multi-service outage.

Detection: lb.healthy_hosts_pct per cluster; lb.no_healthy_upstream count; correlation of check failures across many services at the same moment (a correlated health drop is almost never real).

Mitigation: panic threshold — below 50% healthy, ignore health and use all hosts; if hosts are actually fine, traffic keeps flowing.

Prevention: health endpoints must not call remote shared dependencies; lint for it; dependency failures surface through passive outlier detection on real requests instead. Game day: inject latency in a shared dependency and verify no pool goes empty.

Owner: edge platform (panic threshold, policy); each service team (their /health); the platform that owns the health-endpoint standard.

4.3 The Global Config Push#

t=0:        Route change merged: a new regex path matcher for a marketing page.
            Pattern has catastrophic backtracking on certain paths.
t=+20s:     Config pipeline validates syntax (passes) and pushes globally — no canary
            stage for "route-only" changes.
t=+30s:     Proxy CPU 30% → 100% across every region. Added latency p99 2 ms → 9 s.
t=+1min:    Page: lb.added_latency_p99 > 500 ms, lb.cpu > 90% in all regions.
t=+6min:    On-call identifies the change via config version on dashboards.
t=+9min:    Rollback pushed. Proxies at 100% CPU take 2–3 min to apply it.
t=+13min:   Recovered. 99.99% monthly budget (4.3 min) exceeded 3× in one incident.

A well-known public example of this class: in July 2019, Cloudflare deployed a WAF rule containing a regular expression that caused excessive backtracking; deployed globally at once, it drove CPU to exhaustion across its network for roughly half an hour.

Detection: lb.cpu_utilization, lb.added_latency_p99, config version annotations on every edge dashboard.

Mitigation: one-click rollback to last-known-good; proxies apply config on a separate thread pool or core reservation so a CPU-saturated data plane can still accept a rollback.

Prevention: every config change — including "just a route" — goes through canary → AZ → region → global with bake time; regex engines with linear-time guarantees (RE2-style) for any user-authored pattern; CPU-cost benchmarks of new config in CI.

Owner: edge platform team owns the pipeline and the guardrails; the route author owns the change.

4.4 The TLS Handshake Storm#

t=0:        Scheduled ticket-key rotation runs. A bug distributes the new key to the
            proxies without keeping the previous key for decryption.
t=+0s:      Every resumption attempt fails → full handshakes.
            Resumption rate: 65% → 0%. Full handshakes/s: 50K → 145K.
t=+40s:     Proxy CPU 55% → 97%. Handshake latency p99 50 ms → 1.2 s.
            Mobile clients time out at 10 s, retry → more handshakes.
t=+2min:    Page: tls.full_handshakes_per_sec > 2× baseline, lb.cpu > 90%.
t=+5min:    On-call scales the L7 fleet +40% (takes 4 min to warm).
t=+9min:    Old key restored for decryption. Resumption back to 60%. CPU 60%.

Detection: tls.resumption_rate, tls.full_handshakes_per_sec, tls.handshake_latency_p99, proxy CPU.

Mitigation: restore previous ticket key; scale out; prefer ECDSA (if RSA was serving a share of clients, shifting cuts handshake CPU ~5–10×).

Prevention: ticket keys rotated with overlap (current + previous for decryption); resumption-rate alert at < 50% of baseline; capacity plan for zero resumption at peak, because a key mistake or a mass client reconnect produces exactly that.

Owner: edge platform (cert and key management).

4.5 The Long-Lived Connection Imbalance#

t=0:        Partner gRPC traffic grows. Backend pool scales from 20 to 40 hosts.
t=+10min:   The 20 new hosts serve ~3% of requests. The 20 old hosts at 85% CPU.
            Clients hold HTTP/2 connections for hours; L4 balancing at the backend
            tier balanced once, at connect time.
t=+30min:   Old hosts p99 rises from 80 ms to 900 ms. Autoscaler adds more hosts —
            which also get no traffic.
t=+40min:   On-call restarts old hosts in batches → mass reconnect → spread.

Detection: request distribution skew across hosts (max/mean > 2), new hosts' RPS after scale-out, connection age distribution.

Mitigation: per-request balancing at an L7 hop in front of gRPC backends, or client-side balancing in the gRPC client; max connection age with GOAWAY.

Prevention: never use L4-only balancing for multiplexed protocols; alert on per-host RPS skew; scale-out tests that check new-host traffic share within 5 minutes.

Owner: edge platform (protocol-aware balancing); partner integration team (client config).

4.6 The L4 Rehash#

t=0:        L4 tier scaled from 8 to 10 nodes for an event. Routers' ECMP next-hop
            set changes. A naive mod-N hash would remap ~80% of flows to different
            L4 nodes.
t=+0s:      L4 nodes without shared consistent hashing forward remapped packets to
            different L7 proxies than the ones holding the TCP state → RST.
t=+1s:      ~800K established connections reset. Clients reconnect at once:
            full TLS handshakes spike 10×.
t=+2min:    L7 CPU saturates from the handshake storm (see 4.4).

Detection: tcp.resets_sent at L7, l4.flow_table_misses, sudden handshake spikes coinciding with L4 fleet changes.

Mitigation: consistent hashing (Maglev-style) on every L4 node using the same backend table, so any L4 node sends a given flow to the same L7 proxy; connection tracking for in-flight flows during backend set changes.

Prevention: L4 fleet changes during low-traffic windows; consistent hashing is mandatory; validate in staging that adding an L4 node resets < 1% of flows.

Owner: network / L4 team.

4.7 Operational Reality Matrix#

🎯 Staff Insight: Five of these nine failures are the load balancer amplifying a smaller problem — one bad host, a dependency blip, a key rotation, a fleet change. A good LB design is mostly a list of bounds on its own automated decisions.

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Knowing Maglev's lookup-table math ≠ edge design. Impressive, but the interview is about bounding failures.
• Listing every cloud LB product ≠ judgment. Product names without policies (ejection caps, drain windows) are a catalog, not a design.
• Packet-level depth ≠ L7 understanding. Candidates strong on XDP and DSR sometimes miss request-level health entirely.
• "Use a service mesh" ≠ east-west design. A mesh is a mechanism; the policy (timeouts, retries, outlier rules) is the design.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• OSI model recitation. "L4 sees connections, L7 sees requests" is enough.
• VRRP/keepalived active-passive details. Stateless fleets behind ECMP replace them.
• Exact Maglev table-population algorithm. Name it, state the property (minimal disruption), move on.
• Cipher suite lists. "TLS 1.3, ECDSA with RSA fallback, resumption" covers it.
• WAF rule design. It belongs to an edge-security interview.

6.4 Follow-Up Questions to Expect#

1. "How does a new L7 proxy join without disrupting connections?" — L4 consistent hashing with connection tracking: existing flows stay pinned; only new flows map to the new proxy.
2. "How do you drain a proxy for upgrade?" — Withdraw it from the L4 table for new flows, send GOAWAY / Connection: close, wait for in-flight (≈ p99 request time), cap at a few minutes; WebSockets get reconnect frames.
3. "How do you pick the retry layer?" — Retry where you have the most context and the fewest multipliers: usually the edge for idempotent requests, with a budget; backends don't retry each other's retries.
4. "How do you balance across zones?" — Prefer local zone; spill proportionally when local healthy capacity falls below demand; avoid permanent cross-zone traffic (latency + transfer cost).
5. "What if the control plane is down?" — Proxies keep last-known-good config and endpoint lists; no new routes ship; endpoint churn is absorbed by outlier detection.
6. "How do you protect the LB from DDoS?" — Anycast spreads volumetric attacks; L4 drops malformed packets and SYN floods (SYN cookies); L7 rate limits per client — see Rate Limiting.
7. "How do you roll out HTTP/3?" — Advertise via Alt-Svc to a percentage of clients, compare latency and error rates, keep TCP fallback; UDP needs L4 support for QUIC connection IDs.

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design a load balancer."

Drill 2: The Algorithm#

Prompt: "Why not least-connections?"

Drill 3: The Health Check#

Prompt: "Design the health checking. A teammate wants /health to check the database, cache and auth service."

Drill 4: Make It Concrete — Sizing the Edge#

Prompt: "400K RPS in the largest region, 60K new TLS connections/s, average response 20 KB. Size the L7 and L4 tiers."

Drill 5: The Dependency Goes Down#

Prompt: "A backend service's database slows down. 60% of its hosts start timing out. What does your LB do, minute by minute?"

Drill 6: The Hot Key#

Prompt: "A cache-heavy service uses consistent hashing on user_id for affinity. A celebrity account now generates 8% of its traffic."

Drill 7: Multi-Tenant Edge#

Prompt: "60 services share the edge fleet. One team's route change or traffic spike shouldn't hurt the others. How?"

Drill 8: Build vs Buy#

Prompt: "Cloud-managed load balancer, self-run open-source proxy, or build our own?"

Drill 9: Changing Routing Without an Outage#

Prompt: "Move 30% of traffic for /checkout to a new backend version. Then all of it. Safely."

Drill 10: Multi-Region#

Prompt: "We're going from one region to three. How does global load balancing work, and what happens when a region fails?"

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Incident — Launch Day Handshake Saturation#

Context: A marketing campaign drives a product launch. Requests per second rise 3×, but edge proxy CPU goes from 40% to 98% and handshake latency p99 reaches 2 seconds. Backends are at 50% CPU and fine. The on-call has started adding proxies; it takes 6 minutes per batch to warm up. You're pulled in.

Questions to Surface First:

• New connections per second vs baseline? Is this a request problem or a connection problem?
• What's the resumption rate? It should drop with new users — by how much?
• What fraction of handshakes use RSA vs ECDSA?
• Are clients retrying timed-out handshakes, compounding the load?

Typical L5 Approach: Scales the proxy fleet and waits. Possibly raises proxy worker counts. Recovers in 20–30 minutes once capacity arrives.

Staff Approach: Sees that new users mean near-zero resumption: full handshakes rose 8× while RPS rose 3×. Finds that the cert served is RSA-only because an ECDSA cert was never provisioned for the launch hostname. Deploys the ECDSA cert (canary first), cutting handshake CPU ~5×, while scaling out in parallel. Sets client retry backoff via config for the mobile app's next session.

Principal Approach: Makes "new-connection capacity at zero resumption" a launch-readiness criterion, and moves cert provisioning for new hostnames into the platform with dual ECDSA/RSA as the only supported option. Asks marketing to share campaign calendars with the edge team as part of the launch process.

Key Takeaway: "Launch traffic is new-user traffic, and new users don't resume TLS sessions. Size the edge on new connections at zero resumption, not on requests."

What clears the Staff bar:

• Distinguishes connection growth from request growth
• Finds the cert type as a 5× CPU lever
• Fixes the cause while scaling, not instead of fixing

Deep Dive 2: Silent Failure — One Zone Running Hot for Weeks#

Context: A capacity review shows that one AZ's backends run at 75% CPU while the other two run at 35%. Latency p99 for users routed through that zone has been 40% worse for at least three weeks. No alert fired; overall SLOs were met. Cross-AZ transfer costs also rose 20% last month.

Questions to Surface First:

• Is zone-aware routing enabled? What's the spillover rule?
• Is the L7 fleet itself evenly spread across zones? Are the L4 tier's ECMP weights equal?
• Did a deploy change the number of backends per zone (e.g., one zone got fewer hosts after a capacity shortage)?
• Where does the cross-AZ transfer come from?

Typical L5 Approach: Adds hosts in the hot zone until CPU evens out.

Staff Approach: Finds the mechanism: an instance-type shortage left the hot zone with 12 backends vs 20 in the others, but zone-aware routing sends traffic in proportion to proxy placement, not backend capacity. The hot zone's proxies keep traffic local until hosts are "unhealthy", which they never quite become. Meanwhile, other services had spillover on, creating the cross-AZ transfer. Fixes routing to weight by healthy backend capacity per zone.

Principal Approach: Treats the absence of an alert as the main finding. Adds per-zone utilization skew as an SLO-adjacent signal across all services, and makes "capacity-weighted zone routing" the platform default so zone imbalance self-corrects.

Key Takeaway: "Averages hide zone skew. Route by capacity, not by location alone, and alert on the skew itself."

What clears the Staff bar:

• Finds the mismatch between proxy placement and backend capacity
• Fixes routing policy instead of throwing hosts at it
• Recognizes cross-AZ cost as a second symptom of the same policy

Deep Dive 3: Large-Customer Onboarding — 400K WebSockets From One NAT#

Context: An enterprise customer integrates a real-time dashboard product. They'll open ~400K WebSocket connections, all from a handful of corporate NAT egress IPs, and require their traffic to come from fixed IPs they can allowlist (for callbacks) and mTLS for their API calls. Go-live is in five weeks.

Questions to Surface First:

• Does any layer hash on source IP? If so, a few NAT IPs land on a few proxies.
• What's per-connection memory on the proxies with TLS? 400K × ~30 KB = ~12 GB across the fleet — fine if spread, a problem if concentrated.
• How will these connections drain during deploys — what's the reconnect behavior of their client?
• Is mTLS terminated at the edge with cert validation against their CA, or passed through?

Typical L5 Approach: Adds proxies to absorb the connections and enables sticky sessions by source IP for the WebSockets.

Staff Approach: Avoids source-IP hashing — with a handful of NAT IPs it puts thousands of connections on a few proxies. The L4 tier hashes on the full 5-tuple (source port varies), which spreads well. Places the WebSocket service on a dedicated proxy pool behind the same L4 tier so 400K long-lived connections don't shape the shared fleet. Agrees on a reconnect protocol: server sends a reconnect frame during drains; client reconnects with 0–300 s jitter. mTLS terminated at the edge with their CA in a per-tenant trust bundle; fixed egress IPs via a dedicated NAT for callbacks.

Principal Approach: Uses the deal to define an enterprise-edge tier — dedicated pools, tenant trust bundles, static egress, contractual reconnect behavior — priced into the enterprise SKU rather than built bespoke for one customer.

Key Takeaway: "Source-IP affinity fails exactly for enterprise customers, who arrive through a few NAT addresses. Long-lived connections need their own pool and a negotiated reconnect protocol."

What clears the Staff bar:

• Predicts the NAT hot spot and measures it before go-live
• Isolates long-lived connections from the shared fleet
• Designs draining as a protocol with the client, not a timeout

Deep Dive 4: Post-Mortem — 13 Minutes of Global Latency and 5xx From a Route Change#

Context: You lead the post-mortem for incident 4.3: a route change containing a backtracking regex was pushed globally and saturated every edge proxy for about 13 minutes. Initial root cause: "bad regex." Leadership asks how to make sure it can't happen again.

Questions to Surface First:

• Why did a route change skip canarying? Is there a class of change that's considered "safe"?
• Why could a service team author an arbitrary regex in edge config?
• Why did rollback take 3+ minutes to apply?
• Could the proxies have protected themselves — per-route CPU limits, linear-time regex?

Typical L5 Approach: Root cause: bad regex. Action items: review regexes more carefully; add a regex linter.

Staff Approach: The regex was the trigger. Root causes: (1) route changes bypassed staged rollout because they were classed "low risk"; (2) the regex engine allowed backtracking; (3) config application competed for CPU with the saturated data plane, slowing rollback. Fixes all three: no change class bypasses staging; linear-time regex engine only; config application reserved a core.

Principal Approach: Reframes the edge config pipeline as safety-critical infrastructure. Introduces an error-budget policy: when edge config changes consume more than 25% of the monthly budget, non-emergency edge config changes freeze until the pipeline gains a new safeguard. Reviews every system with global config push in the company — DNS, feature flags, WAF — for the same pattern.

Key Takeaway: "Any change that reaches every proxy at once is a global outage waiting for a trigger. There is no such thing as a safe-to-skip-canary config class."

What clears the Staff bar:

• Refuses "bad regex" as a root cause
• Fixes the pipeline, the engine and the rollback path
• Finds the organizational trade-off that created the exemption

Deep Dive 5: Multi-Region Expansion — Adding APAC#

Context: The company serves APAC users from a US region with ~180 ms RTT. Product wants a Singapore region in two quarters. Backends will be deployed there, but the primary database stays in the US for now. The edge team must decide how traffic reaches the new region and what happens on failure.

Questions to Surface First:

• Which requests can be served fully in-region (reads from replicas, static), and which must cross to the US (writes)?
• Anycast or GeoDNS? Do we have anycast address space and BGP presence in APAC?
• What happens to APAC users if the Singapore region fails — fail to US with +180 ms, or degrade?
• Capacity: can the US region absorb APAC traffic during a Singapore failure?

Typical L5 Approach: Adds a GeoDNS record for APAC pointing to Singapore and deploys the stack there.

Staff Approach: Starts with edge-only: terminate TLS in Singapore first, keep backends in the US, and send requests over warm, pooled connections across the backbone. That alone removes ~2 RTTs × 180 ms of handshake from every new APAC connection — most of the user-visible win — before any backend moves. Then moves read paths into the region, with writes routed to the US by path. Uses GeoDNS initially (no anycast space), with 60 s TTL and failover to the US edge; US capacity planned to absorb APAC at 30% headroom.

Principal Approach: Sequences the investment by user-visible latency per dollar: edge termination first (cheap, large win), read paths second, data residency and writes last (expensive, one-way doors). Ties the plan to the multi-region data strategy so the edge doesn't promise more locality than the data tier can deliver.

Key Takeaway: "Terminating TLS near the user is the cheapest large latency win in a multi-region rollout. Move the handshake first, the reads second and the writes last."

What clears the Staff bar:

• Quantifies handshake RTT savings from edge termination alone
• Sequences the rollout by value and reversibility
• Leaves data-freshness decisions with the owning teams

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Distinguish edge, east-west and L4 load balancing and put each decision at the layer with the right information
• Locate latency correctly: sub-millisecond proxy hops vs round-trip-dominated TLS and TCP setup
• Size an edge fleet on both requests and new connections, including a zero-resumption scenario
• Explain why P2C least-request beats least-connections, and why it still needs outlier detection
• Design health checking that can't empty a pool: shallow active checks, passive ejection, caps and panic mode
• Treat config as a deploy with staged rollout, automatic rollback and data-plane independence from the control plane
• Handle long-lived connections with max age, GOAWAY, per-request balancing and negotiated reconnects
• Plan regional failover as a capacity question with explicit headroom

The Bar for This Question#

Mid-level (L4): Explains what a load balancer does, names round robin and least connections, and adds health checks and a redundant pair. Understands L4 vs L7 in textbook terms.

Senior (L5): Builds a sensible two-tier design with TLS termination, health checks, autoscaling and sticky sessions where needed. Knows the products. Misses how the balancer amplifies failures: herding, sinkholes, correlated health checks, config pushes.

Staff+ (L6): Starts from where latency and failure actually come from. Puts numbers on the TLS cost, chooses P2C with outlier detection and explains why, bounds every automated decision, ships config like code, and handles connection lifetime explicitly. Assigns ownership between the edge platform and service teams. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 The Load Balancer Hop Is Never Your Latency Problem#

The Staff position: Optimize connection reuse, TLS resumption and edge proximity before debating proxy hops.

Why this matters in interviews: Candidates who argue against an L7 tier "for latency" reveal they haven't measured where the time goes.

10.2 Least-Connections Should Be Retired as a Default#

The Staff position: P2C least-request plus outlier detection as the default; least-connections only for a single balancer with uniform, connection-per-request traffic.

Why this matters in interviews: It's the clearest single place where a Senior's reasonable answer has a known production failure mode.

10.3 Deep Health Checks Cause More Outages Than They Prevent#

The Staff position: Health endpoints check the process, not its dependencies. Correctness comes from outlier detection; dependency problems surface as errors, honestly attributed.

Why this matters in interviews: Interviewers who've lived through a correlated ejection outage will recognize this immediately.

10.4 Sticky Sessions Are a State-Placement Bug#

The Staff position: Move session state to a shared store; use bounded-load consistent hashing only as a cache-affinity optimization, never for correctness.

Why this matters in interviews: Pushing back on a requirement and offering the real fix is a Staff behavior interviewers look for.

10.5 Your Edge's Biggest Risk Is Its Config Pipeline#

The Staff position: Invest in staged rollout, automatic rollback and CPU-cost testing of config before the next data-plane optimization.

Why this matters in interviews: It moves the conversation from boxes to change management — where real availability is won or lost.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

The Staff engineer designs a safe edge. The Principal engineer notices that a single request from a user's phone passes through five independently owned balancing decisions: GeoDNS run by the network team, a cloud L4 balancer, the edge proxy fleet, an API gateway owned by another platform team, and a sidecar mesh owned by a third. Each has its own health model, its own retry policy and its own config pipeline. A request can be retried at four layers; a backend can be ejected by three different health systems that disagree. The L7 problem is traffic-management governance: how many layers the org needs, which policies are set once, and who owns the end-to-end behavior of a request that no single team sees whole.

The Org-Level Fault Line#

One traffic platform vs layered teams vs per-service choice.

🧭 Principal Move: "I don't need one team running every proxy. I need one answer to 'how many times can a request be retried, by whom, and with what budget' — and one standard for how config reaches a proxy. The data planes can stay with the teams that know them; the policy and the change pipeline become shared."

Cost Model#

Assumptions: cloud-hosted, 3 AZs per region, fully loaded engineer ~$250K/year, average response 15 KB, ~40% of connections new at peak. Egress is billed separately and usually dwarfs balancer cost; it is excluded from infra but noted.

The pricing insight: at enterprise scale, the biggest controllable line items are not proxies — they're handshake headroom and cross-AZ transfer. Switching to ECDSA with high resumption can shrink the L7 fleet by 30–50%; capacity-weighted zone routing can cut cross-AZ transfer for the edge by half. Each is worth more than any proxy tuning, and each takes one or two engineers a quarter.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-TRAFFIC-001: Traffic Management Standard
Status: Approved   Owners: Edge Platform + Service Mesh Platform + SRE

Scope
  Every balancing layer handling production requests: global steering,
  L4, edge L7, API gateway, service mesh, and client libraries.

MUST
  1. Default balancing is least-request with power-of-two choices. Hash-based
     balancing MUST use bounded load (max 1.25x average).
  2. Passive outlier detection is enabled with max ejection <= 20% of a pool;
     panic threshold is 50% healthy.
  3. Health endpoints used for load balancing MUST NOT call remote shared
     dependencies.
  4. Exactly one layer retries a given request. Retries apply only to
     idempotent requests and are capped by a retry budget of 20% of traffic.
  5. Every config change to any balancing layer rolls out in stages
     (canary, zone, region, global) with automated rollback. No exempt classes.
  6. Data planes continue serving on last-known-good config when the control
     plane is unavailable.
  7. Edge certificates are provisioned by the platform with ECDSA and RSA
     fallback; ticket keys rotate with overlap.

SHOULD
  1. Cap client connection age (10 min +/- jitter) via GOAWAY.
  2. Use capacity-weighted zone-aware routing.
  3. Use dedicated pools for long-lived connection workloads above 50K connections.

Exceptions
  Filed with Edge Platform; reviewed within 5 business days; time-boxed to
  2 quarters; exceptions to MUST 4 and MUST 5 need SRE sign-off.

Success metrics
  - Incidents where a balancing layer amplified a smaller failure: trending to 0
  - Config-change-caused edge minutes of impact: < 25% of edge error budget
  - Edge-added latency p99 (excluding client RTT): < 5 ms
  - TLS resumption rate at peak: > 50%
  - Services with more than one retrying layer: 0

What I'd Tell the VP#

"Our biggest outages at the edge haven't come from machines failing — they've come from our own traffic systems overreacting: a config change that reached every server at once, health checks that removed healthy servers, retries piling up across layers. I'm proposing one shared standard for how all of our traffic layers behave and one safe path for changing them. It's about three engineers for two quarters. We expect it to remove our most common class of large edge incident, and the TLS and zone-routing changes that come with it should reduce edge infrastructure cost by roughly a third. Teams keep ownership of their own systems; what changes is that they follow the same rules."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "I'd configure outlier detection and panic thresholds on the edge" — correct for one layer, but ignores the gateway and mesh making contradictory decisions on the same requests.
• "Retries happen at the edge with a budget" — doesn't ask whether the mesh and client libraries also retry, multiplying the budget.
• "We'll add a region for resilience" — no capacity headroom priced, no failover decision owner, no sequencing by cost.

Appendices

Appendix A: Mechanics in Depth#

A.1 Power of Two Choices with Least Request#

pick_host(cluster):
    hosts = cluster.healthy_hosts()
    if len(hosts) < cluster.panic_threshold * len(cluster.all_hosts()):
        hosts = cluster.all_hosts()                    # panic: ignore health
    a, b = random_sample(hosts, 2)
    score(h) = (h.active_requests + 1) / h.effective_weight()
    return a if score(a) <= score(b) else b

effective_weight(h):
    w = h.configured_weight
    if h.age_seconds < slow_start_window:              # e.g. 60 s
        w = w * max(0.1, h.age_seconds / slow_start_window)
    return w

Why it's right: each proxy needs only its own in-flight counts — no coordination. Randomly sampling two hosts gives near-optimal balance even when counts are stale or partial, and different proxies rarely choose the same host at the same instant, so there's no herd. Why it can go wrong: a host that completes requests instantly (because it fails) always has low in-flight counts. Outlier detection is the necessary partner.

A.2 Outlier Detection#

on_response(host, status, latency):
    if status >= 500 or timed_out:
        host.consecutive_errors += 1
    else:
        host.consecutive_errors = 0
    if host.consecutive_errors >= 5 and can_eject(cluster):
        eject(host, duration = base_ejection * host.times_ejected)   # 30 s, 60 s, 90 s...

every 10 s:  # success-rate outliers, only if pool has >= 5 hosts with >= 100 requests
    mean, stdev = success_rates(cluster)
    for host where host.success_rate < mean - 1.9 * stdev:
        if can_eject(cluster): eject(host)

can_eject(cluster):
    return ejected_count(cluster) < max(1, 0.10 * len(cluster.all_hosts()))

The relative check (vs peers) is what makes outlier detection safe under systemic failure: if every host's success rate drops together, none is an outlier.

A.3 Consistent Hashing Variants#

See Consistent Hashing for the underlying theory.

Appendix B: Routing Keys and Identity#

Precedence rule: the most specific route wins (exact host > wildcard host; longest path prefix), and the config validator rejects ambiguous overlaps at commit time rather than letting proxies decide at runtime.

Appendix C: Coordination Mechanisms#

C.1 Endpoint Propagation#

Service discovery publishes endpoint sets; the control plane pushes incremental updates (deltas, not full snapshots) to proxies over a streaming API (e.g., xDS). Requirements: updates propagate in < 5 s at p99; proxies ACK/NACK each version so the control plane knows which proxies run which config; the control plane batches churn (a 400-host deploy shouldn't produce 400 pushes per proxy). See Service Discovery.

C.2 Quick Comparison#

Appendix D: API Contract & Client Behavior#

Appendix E: Observability#

Core metrics:

Control plane vs data plane. The data plane must keep routing on last-known-good config when the control plane is down; endpoints going stale is tolerable for minutes because outlier detection ejects dead hosts from real traffic. Test it: kill the control plane in staging during a deploy.

Debugging the silent failure. Aggregate dashboards hide per-zone and per-host skew. Always keep: per-host RPS distribution (max/mean), per-zone utilization, and requests by config version. "Which config version served this request?" should be answerable from access logs.

Appendix F: Scale Evolution#

Multi-region path: GeoDNS with health-based failover → edge termination in more regions with backbone to central backends → regional backends for reads → anycast with capacity-aware steering.

What You Don't Build on Day One: your own L4 data plane; anycast; a custom proxy; per-request adaptive algorithms beyond P2C; cross-region active-active steering. Each is justified by a measured problem, not anticipated scale.

Appendix G: Multi-Tenancy, Fairness & Cost#

Per-service limits on shared proxies (prevent one backend from exhausting the proxy):

Cost attribution. Charge services for what they consume at the edge: full handshakes (CPU), requests, egress bytes and long-lived connection-hours. The first report usually shows one or two services dominating handshake cost — typically mobile clients with short keepalives — and fixing their client config pays for the attribution work.