Technologies referenced in this case study: API Gateways & Service Mesh · Redis · Kafka · ZooKeeper & etcd

Related case studies: Rate Limiting · Load Balancer · Service Discovery · API Gateway · Degraded Mode

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once, then come back to the sections where you are weak.

Without fault tolerance:
t=0:      Recommendations service p99 goes from 40ms to 8s (GC death spiral)
t=+5s:    Product-page service has 200 worker threads; all 200 now blocked on recs
t=+10s:   Product-page health checks time out; load balancer marks instances unhealthy
t=+15s:   Remaining instances absorb the traffic, block on recs, go unhealthy too
t=+30s:   Clients retry 3x; product-page ingress QPS is now 4x normal
t=+2min:  Checkout (which calls product-page for prices) starts failing
t=+40min: Full site outage caused by a non-critical widget

With fault tolerance:
t=0:      Same recs slowdown
t=+0.3s:  Product-page's 300ms timeout on recs fires; bulkhead caps recs to 20 threads
t=+5s:    Breaker sees 60% slow calls over 100 requests, opens
t=+5s:    Product page renders with a static 'Popular items' fallback
t=+35s:   Half-open probes: 10 requests; recs still slow; breaker stays open
t=+6min:  Recs team rolls back; probes succeed; breaker closes
Result:   Checkout never noticed. A dashboard blip and a page to the recs team only.

Executive Summary

If you only read one section, read this. Every later section expands one row of what follows.

What This Interview Actually Tests#

Circuit breakers are not a state-machine question. Everyone can draw closed → open → half-open.

This is a cascading-failure containment question that tests:

• Whether you know that the slow dependency, not the dead one, is what kills systems
• Whether you can reason about load amplification — retries, fan-out, and fallbacks that multiply traffic at the worst moment
• Whether you decide, per dependency, what "degraded" means to a user — and who signed off on it
• Whether you put protection in the right place: caller, callee, mesh, or all three

The key insight: A circuit breaker is a policy about which failures you are willing to show users in exchange for keeping everything else alive. The mechanism is 50 lines of code. The policy is a negotiation between product, the calling team, and the dependency team.

The L5 vs L6 vs L7 Contrast — Start Here#

The Staff Positions#

The Three Intents#

"Add circuit breakers" hides three different goals. They lead to different placements, different thresholds and different failure semantics.

🎯 Staff Move: "I'll design for containing the cascade first, because that's what turns a single-team incident into a company outage. But I'll say up front that the breaker alone doesn't let the dependency recover — that needs server-side shedding and retry budgets — and it doesn't decide what users see — that's a per-dependency fallback product has to approve. Three intents, three owners."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Citing specific systems shows you've studied operational reality. Use one of these in the first ten minutes.

Netflix — Hystrix and the Bulkhead Model#

Netflix built and open-sourced Hystrix (2012) after learning that a single slow dependency among hundreds could saturate every request thread in the API tier. Hystrix wrapped each dependency in its own thread pool (a bulkhead — default 10 threads), enforced a timeout, tracked failures in a rolling 10-second window, and opened the circuit when at least 20 requests had been seen and ≥50% failed, probing again after 5 seconds. Every command had an explicit fallback. Hystrix went into maintenance mode in 2018; Netflix moved toward adaptive concurrency limits (their open-source concurrency-limits library) that infer capacity from latency rather than using static thresholds.

Staff insight: The lasting contribution was not the state machine — it was the bulkhead per dependency and the fallback per call. And the move away from static thresholds is the interview-worthy lesson: fixed thresholds are wrong the moment load patterns change.

Google — Client-Side Adaptive Throttling and Criticality#

The Google SRE book describes client-side adaptive throttling: each client tracks requests and accepts over the last two minutes and locally rejects new requests with probability max(0, (requests − K × accepts) / (requests + 1)), with K typically 2. Clients stop sending work the backend is going to reject anyway — a breaker with a continuous, not binary, response. Requests also carry a criticality (e.g. CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE) so an overloaded backend sheds the least important traffic first, plus per-request retry caps (3 attempts) and a per-client retry budget (~10%).

Staff insight: "Circuit breaker" is a special case of adaptive throttling with K → ∞ and a binary output. Citing the formula signals you know the continuous version exists and why it avoids the thundering-herd close.

Amazon — Timeouts, Jittered Retries, and Cells#

The Amazon Builders' Library publishes the company's defaults: timeouts derived from downstream latency percentiles, exponential backoff with jitter, retry token buckets in the AWS SDKs so retries stop when failures are widespread, and load shedding that rejects early rather than letting queued work time out. Amazon's architecture guidance also pushes cell-based architecture and shuffle sharding so a poison request or a bad customer affects a bounded fraction of capacity. The public post-mortem of the September 2015 DynamoDB disruption described exactly the failure this case study is about: storage servers' metadata requests timed out, retried, and kept the metadata service overloaded until load was manually reduced.

Staff insight: Amazon treats retries as a system-level risk, not a client convenience. The DynamoDB event is the canonical example of a metastable failure: the trigger was brief, the retry-driven overload was self-sustaining.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: Protection is layered, and each layer has a different owner. The gateway sets the overall deadline and criticality. The calling service owns bulkheads, breakers and — critically — the fallback semantics, because only it knows what a stale price means. The sidecar (platform team) owns uniform mechanics: connect timeouts, retry budgets, outlier ejection. The callee owns load shedding, because it is the only component that knows its own capacity and cannot trust 400 callers to behave. Observability emits breaker state and fallback counts, because a breaker that is silently open for three days looks exactly like a healthy system on a latency graph.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The prompt usually arrives as one of: "Design fault tolerance for a microservice platform", "Our checkout went down because recommendations were slow — design it so that can't happen", or "Design a circuit breaker library." The third is a trap: if you spend 30 minutes on a state machine you will be leveled Senior. Spend 5 on the mechanism and 30 on placement, amplification, fallbacks and ownership.

Phase 1: Requirements & Framing (2–3 min)#

Say this, nearly verbatim:

"Before I design anything, I want to know the shape of the dependency graph and what 'failure' means to the user. Three questions. First: which dependencies are on the critical path — is there any call the page cannot render without? Second: how deep is the call graph — are we 2 hops or 6? That determines how dangerous retries are. Third: what does the business want when a dependency is sick — an error, stale data, or a partial page? I'll assume a product page with ~8 synchronous dependencies, a 4-deep call graph, 20K RPS at peak, and a 1.5s end-to-end deadline. I'll optimize for containing cascades first."

What you have established in 30 seconds:

• Criticality is per dependency, not global
• Depth determines amplification risk
• Degraded semantics are a product decision
• A deadline exists, and you will propagate it

Phase 2: Core Entities & API (1–2 min)#

Name the entities the design manipulates. Keep it short.

The request context is the API. Everything else is local:

Headers on every internal RPC:
  x-request-deadline: 2026-09-29T10:15:03.412Z   # absolute, not relative
  x-criticality:      CRITICAL | DEGRADABLE | SHEDDABLE
  x-attempt:          1                          # retries set 2, 3...
  x-retry-allowed:    false                      # set by the retrying layer so lower layers don't retry

🎯 Staff Move: "The most important API here is the one nobody draws: the request context. Deadline, criticality and attempt count on every hop are what let a service four layers down make the right decision without knowing the call graph."

Phase 3: High-Level Architecture (≤5 min)#

Staff candidates spend under 5 minutes here. Draw the layered picture from the Executive Summary and name the owner of each layer:

1. Edge/gateway — sets deadline, tags criticality, is the only place user-facing retries happen for non-idempotent flows (usually: none).
2. Calling service — bulkhead per dependency, breaker per dependency, fallback per dependency.
3. Sidecar/mesh — connect timeout, retry budget, outlier ejection, max concurrent requests.
4. Callee — load shedding by queue-wait and criticality; adaptive concurrency limit.
5. Observability — breaker state, fallback served, retry ratio, shed count, deadline-exceeded count.

"That's the whole architecture. The interesting part is not the boxes — it's how the policies interact, especially retries and correlated breaker behavior. Let me go there."

Phase 4: Transition to Depth#

The sentence that steers the interviewer toward your strengths:

"There are three places this design usually fails in production, and I'd like to go through them in order of how often they cause real outages: retry amplification, slow-call detection, and correlated recovery — every breaker in the fleet closing at the same moment. Then I'll cover fallbacks and ownership. Does that order work for you?"

If the interviewer wants the state machine, give it in 90 seconds and pivot back.

Phase 5: Deep Dives (25–30 min)#

Deep dive 1 — Timeouts and deadlines (5 min). Derive, don't guess:

End-to-end deadline:            1,500 ms (set at gateway)
Gateway → product-page:         remaining ≈ 1,480 ms
product-page own work:          ~100 ms
product-page → pricing:         timeout = min(remaining − reserve, p99.9_pricing × 1.5)
                                = min(1,380 − 100, 120 × 1.5) = 180 ms
product-page → recs:            timeout = min(…, p99.9_recs × 1.5) = 300 ms, OPTIONAL
If remaining deadline < 50 ms:  don't call at all — fail/fallback immediately

"If a callee receives a request whose deadline has already passed, it should drop it without doing work. That single check prevents the 'zombie work' that keeps overloaded services overloaded."

Deep dive 2 — Retries and amplification (7 min). State the math, then the policy:

Naive:     each of 4 layers retries 3x  → up to 4^4 = 256 attempts at the bottom per user request
Staff:     retry at 1 layer (the one that knows idempotency), 2 attempts max,
           budget: retries ≤ 10% of successful calls over a 10s window,
           backoff: full jitter, base 25ms, cap 250ms,
           never retry on: deadline exceeded, 429/503 without Retry-After, non-idempotent POST
Result:    worst-case load at the bottom under total failure ≈ 1.1× normal

Deep dive 3 — Breaker trigger and slow calls (5 min). Error-rate breakers miss the dangerous case: a dependency returning 200 OK in 8 seconds. Trip on slow_call_rate too, with a minimum volume so a service getting 3 RPS doesn't flap on one failure.

Deep dive 4 — Correlated recovery (5 min). 500 caller instances each run a local breaker. All see the same failure, all open within ~1s, all go half-open 30s later, all send 10 probes = 5,000 simultaneous probes into a service that just recovered. Fixes: jitter the open duration (30s ± 20%), ramp traffic after close (10% → 25% → 50% → 100% over 60s), and prefer continuous adaptive throttling over binary open/close.

Deep dive 5 — Fallbacks and ownership (5 min). Walk the criticality table; state that product signed off on each fallback; state that fallbacks are exercised weekly in a game day or they are assumed broken.

Phase 6: Wrap-Up (2–3 min)#

"To summarize: timeouts from propagated deadlines, retries at one layer under a 10% budget, bulkheads and breakers per dependency with slow-call detection, fallbacks product has approved, and load shedding in every callee. Mechanics live in the mesh; semantics live in the service. What I'd build next: adaptive concurrency limits to replace hand-tuned thresholds, a criticality registry the mesh reads, and a monthly game day that forces every optional dependency's breaker open in production for 10 minutes. The biggest remaining risk is fallbacks that have never run — I'd measure that as 'days since fallback last served traffic' per dependency."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Senior engineers own services. Staff engineers own the interactions between services. Fault tolerance is the canonical interaction problem: every individual component can be well-built, and the system can still collapse because of how they combine — retries that multiply, timeouts that nest the wrong way, breakers that synchronize, fallbacks that shift load onto the database that was already struggling. No single team's code review catches any of it.

That is why interviewers use it. It separates candidates who think "my service is robust" from candidates who think "my service's behavior under failure is part of someone else's failure."

1.2 The L5 vs L6 vs L7 Contrast — Visual#

The L5 path is not wrong — a breaker on recs would have helped. It fixes this incident. The L6 path asks why the architecture allowed it, and fixes the class. The L7 path makes the fix the default for the next 300 services.

1.3 The Staff Question That Cuts Through Everything#

"When this dependency is slow — not down, slow — what happens to the caller's threads, and what does the user see?"

Ask this for every dependency on the whiteboard. It forces:

• A timeout (otherwise the answer is "threads block forever")
• A bulkhead (otherwise the answer is "all threads block")
• A fallback (otherwise the answer is "an error page")
• A named owner for the fallback semantics

If you only remember one sentence from this case study, remember that one.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Intent 1: Contain the cascade (protect the caller). The failure you are preventing is resource exhaustion in the caller: threads, connections, memory for queued requests, and latency budget. By Little's Law, a caller at 1,000 RPS whose dependency goes from 50ms to 5s needs 5,000 in-flight slots instead of 50. No thread pool survives that. The tools are timeouts, bulkheads and caller-side breakers. Success metric: the caller's own p99 and availability stay inside SLO while the dependency is at 0%.

Intent 2: Let the dependency recover (protect the callee). The failure you are preventing is a metastable state: the dependency could recover if load dropped, but retries and queued work keep it saturated. Caller-side breakers help if every caller has one — but you don't control every caller. The tools are server-side load shedding (reject early and cheaply), retry budgets, and adaptive concurrency. Success metric: under 3× offered load, the callee's goodput stays near capacity instead of collapsing to zero.

Intent 3: Preserve the user experience (degrade gracefully). The failure you are preventing is a binary outcome — full page or error page — when a partial page would do. The tools are fallbacks: omit the section, show a static default, serve cached data with a staleness cap, or accept the write and process later. Success metric: conversion or task completion during a dependency outage, measured against baseline.

🎯 Staff Move: "These three have different owners. The caller team owns containment, the dependency team owns their own recovery, and product owns what the degraded experience is. If I design only the breaker, I've only designed the first one."

2.2 When NOT to Use Circuit Breakers#

Breakers are not free. They add a state machine, thresholds that need tuning, and a code path (the open state) that almost never runs. Don't add one when:

"The best circuit breaker is a dependency you took off the synchronous path."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

These are the tensions where good engineers disagree. For each: options, who pays, the Staff default, and when to deviate.

3.1 Fault Line 1: Fail Fast vs Degrade#

When the breaker is open, you either return an error or serve something else.

The Staff default: Classify first. Optional → omit. Degradable → cached with an explicit staleness cap and a visible indicator if the staleness matters. Critical → fail fast with a clear message; never fake success.

The trap: Fallbacks that call another dependency. "If the cache is down, fall back to the database" converts a cache outage into a database outage — the database was sized for a 5% miss rate and now takes 100%. A fallback must be cheaper and more available than the primary, or it is not a fallback.

🎯 Staff Move: "Every fallback I propose has three properties: it's cheaper than the primary, it doesn't depend on anything the primary depends on, and product has signed off on what it shows. A fallback that fails any of those is a second outage waiting to happen."

When to deviate: For read-heavy catalogs where staleness is harmless (product descriptions, images), serve stale aggressively — even hours old. For anything involving money, entitlements, or authorization, fail fast; a stale "allow" is a security incident.

3.2 Fault Line 2: Caller Protection vs Callee Protection#

Breakers live in the caller. Load shedding lives in the callee. Which do you rely on?

The Staff default: Both, with the callee's shedding as the guarantee and the caller's breaker as the optimization. A service that depends on its callers' good behavior for survival does not have fault tolerance.

What callee-side shedding looks like:

on request arrival:
  if now() > request.deadline:                 drop, no response work   # zombie
  if queue_wait_p50 > 50ms and criticality == SHEDDABLE:  reject 503 + Retry-After
  if inflight > adaptive_limit:                reject by lowest criticality first
  else: admit
cost of rejection must be ≤ ~1–5% of cost of serving

When to deviate: In a small system with 2–3 callers you control, caller-side breakers alone are acceptable for a first version — say so explicitly and name the trigger (first external or legacy caller) for adding shedding.

3.3 Fault Line 3: Retry for Success vs Retry Amplification#

Retries are the most dangerous fault-tolerance feature because they work perfectly in testing.

The Staff default: Retries only at one layer (usually the one closest to the failing dependency, or the sidecar), only for idempotent operations, full jitter, and a budget. Signal "don't retry" downstream with a header.

retry_allowed(req, resp):
  if not req.idempotent:                     return false
  if resp.status in {400..499} except 429:   return false
  if deadline_remaining(req) < min_useful:   return false
  if retries_10s / successes_10s > 0.10:     return false   # budget exhausted
  return true
backoff = random(0, min(cap=250ms, base=25ms * 2^attempt))    # full jitter

🎯 Staff Move: "Three retries is a good policy for one client and a terrible policy for a fleet. I'd replace 'max attempts' with a budget: retries can add at most 10% to the load. When failures are rare, that's plenty; when they're widespread, retries disappear automatically, which is exactly when we need them to."

When to deviate: At the very edge (mobile client over a flaky network), one retry on connection failure is almost always right — the failure is likely the network, not the server. Hedging is worth it for read paths with strict tail latency SLOs (search, ads), with its own budget.

3.4 Fault Line 4: Static Thresholds vs Adaptive Limits#

The Staff default: Static breakers with a minimum call volume and slow-call detection as a starting point, bulkhead sizes derived from Little's Law, and a roadmap item to move to adaptive concurrency on the highest-traffic edges. Say the math:

bulkhead size ≈ peak RPS to dependency × p99 latency × headroom
             = 400 RPS × 0.080 s × 1.5 ≈ 48 → round to 50
When the dependency slows to 1s: 400 × 1.0 = 400 needed > 50 → 350 RPS rejected fast
That rejection is the feature: it caps the damage at the bulkhead.

When to deviate: Go adaptive first when traffic is highly variable (10× diurnal swing) or the dependency's capacity changes with autoscaling — static thresholds will be wrong half the day.

3.5 Fault Line 5: Library vs Mesh (Ownership)#

The Staff default: Split by concern. Mechanics — connect timeouts, retry budgets, outlier detection, max concurrency — in the mesh, configured by platform with per-service overrides reviewed in code. Semantics — per-call timeouts tied to the deadline, fallbacks, criticality — in the service, via a thin, standard library that emits standard metrics.

The critical rule: Retries must be configured in exactly one of the two. The most common real-world amplification bug is a library that retries 3× sitting on top of a sidecar that also retries 3× — 16 attempts per call, and neither team knows.

🎯 Staff Move: "The mesh owns how we fail. The service owns what the user sees when we do. And retries live in the mesh only — I'd have the service library refuse to retry if it detects a sidecar."

When to deviate: No mesh and fewer than ~20 services in 1–2 languages: a standard library is fine. Say what would trigger the move (a third language, or the first incident caused by library version drift).

4. Failure Modes & Operational Reality#

The breaker state machine itself, for reference — then the ways the whole system actually fails.

4.1 Retry Storm → Metastable Failure — Full Timeline#

The most common way fault-tolerance features cause an outage.

Setup: 4-layer call graph (gateway → BFF → orders → inventory), each layer retries 3x
       on timeout. inventory capacity: 12,000 RPS. Normal load: 8,000 RPS.

t=0:       inventory DB failover; inventory p99 goes 30ms → 2s for 20 seconds
t=+1s:     orders' 500ms timeouts fire; orders retries 3x → inventory offered 32,000 RPS
t=+2s:     BFF's calls to orders time out; BFF retries 3x → orders offered 4x load,
           each of which fans out 4x to inventory → inventory offered up to 128,000 RPS
t=+20s:    DB failover completes. inventory is healthy — but offered 10x its capacity
t=+25s:    inventory queues grow; every request waits > timeout; all work is wasted
           goodput → ~0 even though every inventory host is "up"
t=+5min:   on-call scales inventory 2x; new hosts are immediately saturated
t=+18min:  someone disables retries in BFF via config push; load drops to 2x
t=+22min:  inventory drains queues and recovers
Trigger lasted 20 seconds. Outage lasted 22 minutes.

Detection: rpc.retry_ratio (retries ÷ first attempts) jumping from ~0.01 to > 1.0; inventory.goodput diverging from inventory.throughput; deadline_exceeded_total rising at the callee while CPU is pegged.

Mitigation (now): Kill-switch retries fleet-wide via mesh config (should take < 60s). Enable aggressive load shedding at inventory (drop requests whose deadline has passed — this alone often breaks the loop).

Prevention: One retry layer; retry budgets; callee drops expired-deadline work; a documented, tested fleet-wide "retries off" switch.

Owner: Platform team owns the retry policy and the kill switch. Inventory team owns shedding. The incident commander owns the decision to flip the switch — pre-approved in the runbook.

4.2 The Slow Dependency That Never Trips the Breaker#

t=0:      recs-svc starts returning 200 OK in 6–9 seconds (a lock contention bug)
t=+10s:   product-page's breaker is error-rate based: 0% errors → stays CLOSED
t=+15s:   product-page has a 10s timeout (the library default nobody changed)
t=+20s:   all 200 request threads blocked on recs; product-page latency 9s
t=+30s:   LB health checks to product-page time out → instances marked unhealthy
t=+45s:   product-page fully down. recs is "healthy" by every dashboard.

Detection: breaker.slow_call_rate, threadpool.active / threadpool.max per bulkhead, dependency.latency_p99 vs its SLO.

Mitigation: Emergency config: drop recs timeout to 300ms, force recs breaker open.

Prevention: Slow-call threshold as a first-class trip condition (Resilience4j supports slowCallDurationThreshold / slowCallRateThreshold); timeouts derived from p99.9; bulkhead for every non-critical dependency so exhaustion is local; a lint rule that fails the build on library-default timeouts.

Owner: product-page team (their timeout, their bulkhead). Platform owns the lint rule.

🎯 Staff Move: "Error-rate breakers protect you from dead dependencies, which were already cheap. Slow-call breakers protect you from sick ones, which are the ones that kill you."

4.3 Correlated Breakers — The Synchronized Herd#

Why it happens: Identical thresholds, identical open durations, identical failure signal. Independent breakers become a synchronized oscillator.

Detection: breaker.state_transitions_total spiking at a fixed period; callee load graph showing a sawtooth with period = open duration.

Prevention: Jitter the open duration (±20%); limit half-open probes per instance (1–5) and ramp traffic after close (10% → 25% → 50% → 100% over 30–60s); prefer continuous client-side adaptive throttling so there is no cliff; callee warms caches before advertising healthy.

Owner: Platform (breaker defaults). Callee team (warm-up before readiness).

4.4 The Fallback That Became the Outage#

t=0:      Redis cache cluster for user profiles loses a shard
t=+1s:    profile-svc breaker on Redis opens; fallback = "read from Postgres"
t=+2s:    Postgres, sized for a 3% cache-miss rate (~600 QPS), receives 20,000 QPS
t=+10s:   Postgres connection pool exhausted; p99 30s; replicas fall behind
t=+30s:   Every service that reads Postgres directly (billing, auth) starts failing
Cache incident became a database incident became a company incident.

Detection: fallback.served_total{dependency} by fallback target; db.connections_in_use / max.

Prevention: Fallback targets must be sized for fallback load or rate-limited (e.g., fallback to DB capped at 2× normal miss traffic, the rest served a static default); request coalescing on miss; see Distributed Caching for thundering-herd controls.

Owner: profile-svc team — they chose the fallback. The DB team should have been consulted on the fallback's load; that consultation is the organizational fix.

4.5 The Breaker That Hid a Bug#

A deploy introduces a bug that makes 15% of calls to the tax service fail with 400 (bad request). The breaker counts 4xx as failures, opens repeatedly, and the fallback (a flat "estimated tax") serves quietly for two weeks. Finance discovers under-collected tax at month-end.

Lessons: Don't count caller-caused 4xx as dependency failures. Alert on fallback.served_ratio sustained above baseline for > 15 minutes. A fallback serving for days is an incident, not a success.

Owner: Calling team for classification; product/finance for the decision that estimated tax was ever an acceptable fallback.

4.6 Silent Open — Nobody Knows the Breaker Is Open#

The breaker's whole purpose is to hide failures from users. Which means it also hides them from dashboards that measure user-facing error rate. Mandatory metrics:

breaker_state{caller, dependency}                  gauge: 0 closed, 1 half-open, 2 open
breaker_open_seconds_total{caller, dependency}     counter
fallback_served_total{caller, dependency, kind}    counter
fallback_last_served_timestamp{dependency}         gauge — "days since fallback ran"

Alert: any CRITICAL-tier dependency breaker open > 60s → page caller owner and dependency owner. Any DEGRADABLE breaker open > 15 min → ticket + notify product.

4.7 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Knowing Hystrix configuration keys ≠ understanding fault tolerance. Configuration trivia signals you've used a library, not that you've debugged a retry storm.
• Drawing a service mesh ≠ solving ownership. The mesh can't write a fallback.
• "We'll use chaos engineering" ≠ resilience. Chaos without hypotheses, blast-radius limits and owners is just random outages.
• A distributed, consistent breaker state store ≠ sophistication. It adds a dependency to your fault-tolerance layer — the one layer that must not have dependencies.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Library selection debates. One sentence.
• Distributed breaker state. Mention and reject.
• Exact sliding-window data structure. Ring buffer of buckets; move on unless asked.
• Every possible fallback type. Show the criticality table with three rows; that's enough.

6.4 Follow-Up Questions to Expect#

1. "What's the difference between a circuit breaker and a rate limiter?" — Breaker reacts to the dependency's health; limiter enforces a client's policy regardless of health.
2. "How do you pick the breaker thresholds?" — Minimum volume from traffic (≥ 20–100 calls per window), slow-call duration from p99.9, error rate starting at 50%, then validate with fault injection.
3. "Should 4xx errors trip the breaker?" — No, except 429. They're caller errors, not dependency health.
4. "How do you know a fallback works?" — It serves production traffic regularly: game days or a small permanent percentage.
5. "Where do retries live if we have a mesh and a library?" — One place — the mesh — with the library's retries disabled.
6. "How does the callee tell callers to back off?" — 503/429 with Retry-After, and gRPC status + pushback metadata; callers' budgets honor it.
7. "What happens when half the fleet has a new breaker config and half doesn't?" — Staged config rollout with canary and automatic rollback on deadline-exceeded or fallback rate.

7. Active Drills#

Drill 1: The Opening#

Prompt: "Our site went down last week because the recommendations service got slow. Design fault tolerance so it doesn't happen again."

Drill 2: Retry Math#

Prompt: "Every service in our call graph retries 3 times with exponential backoff. Is that a problem?"

Drill 3: Make It Concrete — Timeouts#

Prompt: "What timeout would you put on the call from product-page to pricing?"

Drill 4: The Dependency Is Down — What Does the User See?#

Prompt: "Pricing is down. The breaker is open. What does the product page show?"

Drill 5: Callee Can't Trust Callers#

Prompt: "Our inventory service has 40 callers. Some are legacy services with aggressive retries and no breakers. How do you protect inventory?"

Drill 6: The Synchronized Herd#

Prompt: "We have 600 instances of the checkout service, each with a breaker on the fraud service. After a fraud outage, fraud recovered but immediately fell over again, three times. Why?"

Drill 7: Library or Mesh?#

Prompt: "We have 250 services in Java, Go, Node and Python. Some use Resilience4j, some Polly-style ports, some nothing. What should we standardize on?"

Drill 8: Change a Threshold Without an Outage#

Prompt: "You want to lower the default request timeout in the mesh from 15s to 2s for all services. How do you roll it out?"

Drill 9: Cost#

Prompt: "Leadership asks: we spend a lot on resilience features. Is it worth it?"

Drill 10: Multi-Region#

Prompt: "We're active-active in two regions. If a dependency in us-east is failing, should callers fail over to us-west's copy?"

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Cascade#

Context: It's the first hour of a major sale. Traffic is 6× normal. The search service's p99 went from 80ms to 1.2s, and within four minutes the home page, category pages and cart are all erroring at 30–40%. Checkout is at 12% errors. The incident commander pulls you in.

Questions to Surface First:

• Is search slow or failing? Is its CPU saturated, or is it waiting on its own dependency?
• Which callers of search have bulkheads and breakers, and are any breakers open?
• What is the fleet-wide retry_ratio right now versus baseline?
• Why is cart failing — does cart call search, or is cart sharing a thread pool or database with something that does?

Typical L5 Approach: Scales search horizontally, bumps its thread pool, and restarts unhealthy instances. Reasonable — and it treats search as the problem, when the outage is really everyone else failing because of search. New search instances saturate instantly because offered load includes retries.

Staff Approach: Contains first, fixes second. Force-opens the search breakers in non-critical callers (home page carousels, category "related searches") so those pages render with fallbacks, freeing threads immediately. Cuts retries to search to zero via mesh config. Asks search to enable shedding of SHEDDABLE traffic (autocomplete, prefetch) to reserve capacity for the actual search results page. Then scales search, now that scaling can actually help.

Principal Approach: Asks why a sale — a scheduled, predictable event — found search at 6× load without a pre-scaled, load-tested posture. Institutes an event-readiness review for tier-1 paths (load test at 1.5× forecast, verified fallbacks, pre-approved force-open list). Looks at the dependency graph that let cart fail because of search, and makes "no optional dependency shares a resource pool with a critical path" an architecture-review rule.

Key Takeaway: "In a cascade, contain first — free the callers — then fix the dependency. Scaling a service whose offered load includes a retry storm is pouring water into a bucket with no bottom."

What clears the Staff bar:

• Treats callers' resource exhaustion as the outage, not the slow dependency
• Turns off retries before scaling
• Has a pre-approved force-open list and uses it

Deep Dive 2: The Silent Fallback#

Context: Finance reports that shipping-cost revenue is 9% below forecast for the month. Investigation shows the checkout service's breaker on the shipping-rates service has been opening intermittently for 23 days, and the fallback — a flat $5.99 rate — has served ~18% of orders. No alert fired. User-facing error rate was 0%.

Questions to Surface First:

• Why did the breaker open — is shipping-rates actually unhealthy, or is the breaker counting something it shouldn't (4xx, a new slow endpoint)?
• Who approved "flat $5.99" as a fallback, and was it approved as a brief-outage measure or indefinitely?
• Why is there no alert on fallback rate?
• Are there other breakers in the same state right now?

Typical L5 Approach: Fixes the root cause in shipping-rates (it turned out a new carrier integration made 20% of calls take 2.5s, over the 2s slow-call threshold), closes the ticket. Correct, but leaves the systemic hole: any breaker can serve a fallback indefinitely without anyone knowing.

Staff Approach: Fixes the root cause, then fixes observability and policy. Adds fallback_served_ratio alerting per dependency with a duration threshold. Adds a "max fallback duration" to each fallback's definition: flat-rate shipping is acceptable for 30 minutes, after which it pages. Audits all breakers fleet-wide for sustained open or fallback states — finds two more.

Principal Approach: Reframes fallbacks as business policy with an expiry. Every fallback in a revenue- or compliance-adjacent path gets a registered owner in product/finance, a maximum duration, and an estimated $/hour cost. The fallback registry becomes an input to monthly business reviews. The org learns that "0% errors" is not the same as "working."

Key Takeaway: "A breaker's job is to hide failures from users. Your job is to make sure it doesn't hide them from you."

What clears the Staff bar:

• Treats a sustained fallback as an incident, not a success
• Adds duration limits and business owners to fallbacks
• Audits the fleet for the same class of problem

Deep Dive 3: Onboarding a Large Internal Caller#

Context: The data-platform team wants to call the user-profile service from a new batch enrichment job: 50M lookups nightly, ideally in 2 hours (~7,000 RPS). Profile serves 15,000 RPS of interactive traffic at peak with ~35% headroom at night. The profile team asks you to review.

Questions to Surface First:

• Can the batch job read from a replica, a snapshot, or a CDC-fed copy instead of the online service?
• If it must call online: what criticality will its requests carry, and what does it do when shed?
• What's profile's actual nighttime headroom in RPS, and what happens if an interactive spike coincides with the batch?

Typical L5 Approach: Adds a rate limit for the batch caller at 7,000 RPS and a breaker in the batch job. Works on a normal night — and on a bad night the batch still consumes 7,000 RPS of capacity while interactive users suffer.

Staff Approach: Prefers moving the batch off the online path (a nightly snapshot export or a CDC-fed read store). If it must be online: tag all batch requests SHEDDABLE, give the batch a per-caller concurrency quota, and use adaptive client-side throttling so it slows automatically when profile rejects. The batch deadline (2 hours) is soft — it's fine if it takes 3 hours on a busy night. Interactive traffic always wins.

Principal Approach: Establishes a rule for the company: online services are not bulk data sources. Batch consumers get data through the platform's snapshot/CDC path, which the data platform funds. Prevents the next ten teams from making the same request, and removes a whole class of capacity conflicts.

Key Takeaway: "Criticality tags turn 'who wins under contention' from an incident-time argument into a design-time decision."

What clears the Staff bar:

• Prefers removing the coupling over protecting it
• Makes the batch shed first by construction
• States who bears the risk of the batch running late

Deep Dive 4: Post-Mortem — The Mesh Config That Took Down Everything#

Context: A platform engineer pushed a mesh config change intended to lower the default retry count. A typo set per_try_timeout to 10ms fleet-wide. Within 90 seconds, ~70% of inter-service calls were timing out. Rollback took 14 minutes because the config pipeline itself depended on a service that was now failing. You're running the post-mortem.

Questions to Surface First:

• Why did a fleet-wide change go to 100% at once?
• Why did no validation catch a 10ms timeout?
• Why did the rollback path depend on the data plane it controls?
• What was the blast radius by tier, and did any CRITICAL path have a safeguard?

Typical L5 Approach: Adds input validation for timeout values (min 50ms). Necessary, insufficient — the next bad config will be a different field.

Staff Approach: Treats config as code with a deployment pipeline: schema validation plus semantic checks (no timeout below observed p50 for that route), shadow evaluation, staged rollout by tier and region with automatic rollback on deadline_exceeded_total, and a rollback path that works when the mesh is broken (control plane talks to proxies over a path that doesn't traverse the proxies' own routing, and the last-known-good config is retained locally).

Principal Approach: Recognizes the resilience platform is itself the largest correlated-failure risk in the company — one config reaches every service. Requires cell-based rollout for all platform config (cell 1 → wait → cell 2…), tracks "fleet-wide changes per quarter" as a risk metric, and gives the platform team an error budget of its own. The fault-tolerance layer must be more conservative than anything it protects.

Key Takeaway: "The component that protects every service is the component that can break every service. Roll it out like it."

What clears the Staff bar:

• Identifies the circular dependency in recovery tooling
• Proposes a pipeline, not a validation rule
• Applies blast-radius thinking to the resilience platform itself

Deep Dive 5: Multi-Region Expansion#

Context: The company is going from one region to two, active-active. The resilience design so far is regional. Leadership asks: "If a service fails in one region, can we just send its traffic to the other?"

Questions to Surface First:

• What's each region's utilization at peak? Can either absorb the other's load?
• Are deploys staggered across regions, or can a bad release break both?
• Which data is regional and which is global — does cross-region failover of a service also mean cross-region data access?
• What is the cross-region RTT, and how many hops would a cross-region failover add to a request?

Typical L5 Approach: Configures the mesh to fail over to the other region's instances when local instances are unhealthy. Sounds robust; creates a path by which one region's failure doubles load on the other region's copy — correlated global failure.

Staff Approach: Contain within region; fail over at the edge for whole-user traffic, not per-dependency. Per-dependency cross-region failover only for a small allowlist of stateless, idempotent read services with verified headroom, and capped (e.g., ≤ 20% of the remote capacity). Staggered deploys (region A, bake 1 hour, region B) so both copies are rarely broken at once. Capacity: each region provisioned to take ~100% of global peak for failover, or explicitly accept brownout with shedding.

Principal Approach: Decides the company's failure-domain strategy: regions as blast-radius boundaries with cells inside them. Prices the headroom — two regions at 50% utilization cost ~2× the compute of one at 100% — and puts that choice to leadership as an availability-vs-cost decision with numbers. Standardizes "failover is an edge decision" so 200 service teams don't each invent cross-region fallbacks.

Key Takeaway: "Failover is load transfer. If the destination can't take the load, you've built a mechanism for global outages."

What clears the Staff bar:

• Treats cross-region failover as a capacity decision
• Moves failover to the edge, whole-user granularity
• Staggers deploys to break correlation

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Explain why a slow dependency is more dangerous than a dead one, using Little's Law with numbers
• Compute worst-case retry amplification for an N-layer graph and design a retry budget that bounds it to ~1.1×
• Derive a timeout from p99.9 latency and a propagated deadline
• Size a bulkhead from RPS × latency × headroom
• Classify dependencies as critical / degradable / optional and specify a product-approved fallback for each
• Explain why callees must shed load independently of caller breakers, and how criticality ordering works
• Diagnose synchronized breaker oscillation and fix it with jitter, probe limits and ramp-up
• Split fault-tolerance ownership among platform (mechanics), calling team (semantics) and callee team (shedding)
• Explain why the resilience platform's own config is the biggest correlated-failure risk, and how to roll it out safely

The Bar for This Question#

Mid-level (L4): Knows the breaker states and can implement one. Mentions timeouts and exponential backoff. Designs for the dependency being down.

Senior (L5): Configures breakers, timeouts and retries sensibly for a single service. Mentions fallbacks and bulkheads. Misses amplification across layers, slow-call detection, correlated behavior, and who owns fallback semantics.

Staff+ (L6): Starts from dependency criticality, designs for slowness, quantifies retry amplification and bounds it with budgets, puts protection on both sides of every call, makes fallbacks a product decision with sign-off and duration limits, anticipates correlated recovery, and splits ownership between platform and product. Closes with how the system is tested (game days) and what can still go wrong. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "The Circuit Breaker Is the Least Important Part of Fault Tolerance"#

The Staff position: Build timeouts, bulkheads, budgets and shedding first. The breaker is an optimization on top that saves wasted work and gives a clean point to attach fallbacks.

Why this matters in interviews: Leading with the breaker state machine signals you've memorized the famous pattern. Leading with timeouts and amplification signals you've been paged.

10.2 "Most Fallbacks Are Untested Code That Will Fail When You Need It"#

The Staff position: A fallback that hasn't served production traffic in the last 30 days is assumed broken. Exercise them in game days or route a small permanent percentage through them.

Why this matters in interviews: Saying "and we'd verify the fallback runs monthly in production" is a single sentence that separates people who design from people who operate.

10.3 "Retries Cause More Outages Than They Prevent — At the Fleet Level"#

The Staff position: Retries are fine at one layer, with a budget. Unbudgeted retries at every layer are the single most common amplifier in large-scale outage post-mortems — the September 2015 DynamoDB event and the Metastable Failures research (Bronson et al., HotOS 2021) both describe retry-sustained overload.

Why this matters in interviews: It's a counterintuitive claim you can defend with arithmetic in 20 seconds.

10.4 "Don't Share Breaker State Across Instances"#

The Staff position: Keep breakers local. If you want a fleet-level signal, get it from the callee (503 + Retry-After, load reports) or from the mesh control plane's outlier data — not from a shared store in the hot path.

Why this matters in interviews: Interviewers often suggest shared state to see whether you'll add a dependency to your resilience layer. Declining with a reason is the signal.

10.5 "Service Mesh Breakers Aren't Circuit Breakers"#

Mesh "circuit breaking" (Envoy's max_connections, max_pending_requests, max_requests) is a set of concurrency caps, closer to a bulkhead. Outlier detection ejects hosts, not services. Neither serves a fallback. Teams that "turned on circuit breaking in Istio" often believe they have Hystrix-style protection and fallbacks when they have neither.

The Staff position: Use the mesh for caps and host ejection; you still need service-level semantics in code.

Why this matters in interviews: Precision about what the mesh does and doesn't do is a fast credibility signal.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer makes one call graph survive one slow dependency. A Principal engineer notices that the company has 400 services, 3,000 dependency edges, four resilience libraries, one mesh, and no shared definition of what "critical" means — and that the next large outage will come not from a missing breaker but from a correlated event: a mesh config push, a shared database, a common library bug, a region-wide deploy. At L7, fault tolerance is a portfolio of blast-radius bets, priced in outage minutes and engineer-years, and the most leveraged decisions are the ones that make good behavior the default for teams who will never read this case study.

The Org-Level Fault Line#

Centralized resilience platform vs per-team autonomy.

🧭 Principal Move: "I'd centralize mechanics and the criticality taxonomy, and deliberately not centralize fallbacks. The day the platform team starts writing product fallbacks is the day it becomes the bottleneck for every launch."

Cost Model#

Assumptions: mesh sidecar ~0.1–0.25 vCPU and ~50–100MB per pod at moderate load; blended compute ~$30/vCPU-month; fully loaded engineer ~$300K/year; revenue at risk scales with company size. Rough, order-of-magnitude.

The biggest line item at scale isn't compute — it's headroom. Running every tier-1 service at ≤ 60% utilization so it survives recovery surges and regional failover can cost 30–70% more than running at 85%. That is the decision leadership actually has to make.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Inter-Service Fault Tolerance Standard v1

Scope: All synchronous RPC between production services. Asynchronous messaging is covered by the Messaging Standard.

Requirements:

• Every request MUST carry an absolute deadline and a criticality (CRITICAL / DEGRADABLE / SHEDDABLE). The gateway sets defaults.
• Every outbound call MUST have a timeout ≤ remaining deadline. Library default timeouts are prohibited (lint-enforced).
• Retries MUST occur in the mesh only, for idempotent methods, with a budget ≤ 10% of successful requests per destination. Application-level retries require an exception.
• Every service MUST drop requests whose deadline has passed and SHOULD shed SHEDDABLE traffic when queue wait exceeds its SLO.
• Every DEGRADABLE or OPTIONAL dependency MUST have a registered fallback with an owner, maximum duration, and product sign-off.
• Every tier-1 fallback MUST serve production traffic at least once per 30 days (game day or permanent sampling).
• Services MUST emit breaker_state, fallback_served_total, retry_ratio, shed_total{criticality}.

Exceptions: Filed with the platform team, reviewed by the architecture group, expire after 6 months unless renewed with data.

Success metrics: Cascading incidents (≥ 2 services affected by one trigger) per quarter; fleet retry_ratio p99 < 0.1; % of tier-1 fallbacks exercised in 30 days = 100%; exception count trending down.

What I'd Tell the VP#

"Most of our big outages last year started as a small problem in one service and spread because of how our services retry and wait on each other. We're going to make the platform handle that by default: every request carries a deadline, retries are capped centrally, and every service can say 'no' when it's overloaded. Product teams will decide what customers see when a feature is degraded, and we'll test those decisions monthly instead of discovering them during incidents. It costs about four platform engineers and a few percent more compute. The goal is that next year's incidents stay the size of one feature, not the size of the company."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• A perfect per-service design with no plan for the other 399 services.
• "We'll add chaos engineering" without abort criteria, owners, or a metric that shows it's working.
• Ownership boundaries named for one system, but no view on how the platform team is funded or what it refuses to do.

Appendices

Appendix A: Mechanics in Depth#

A.1 Sliding-Window Breaker (Count-Based)#

class Breaker:
  state = CLOSED
  window = RingBuffer(size=100)          # last 100 outcomes: SUCCESS / FAILURE / SLOW
  min_calls = 20
  failure_threshold = 0.50
  slow_threshold = 0.50, slow_call_ms = 200
  open_ms = 30_000 * uniform(0.8, 1.2)   # jitter
  half_open_permits = 5

  def allow():
    if state == OPEN and now() - opened_at >= open_ms: state = HALF_OPEN; permits = half_open_permits
    if state == OPEN: return False
    if state == HALF_OPEN: return permits.try_acquire()
    return True

  def record(outcome, latency_ms):
    if outcome == CALLER_ERROR_4XX: return          # not dependency health (except 429)
    o = SLOW if latency_ms > slow_call_ms and outcome == SUCCESS else outcome
    window.add(o)
    if state == HALF_OPEN: evaluate_probe(o); return
    if window.count() >= min_calls:
      if window.rate(FAILURE) >= failure_threshold or window.rate(SLOW) >= slow_threshold:
        state = OPEN; opened_at = now()

Why count-based vs time-based: count-based windows adapt to traffic (100 calls is 1s at peak, 1 minute at night). Time-based (e.g., 10 × 1s buckets) gives consistent reaction time but needs a minimum volume guard. Use time-based for high-traffic edges, count-based with min volume otherwise.

A.2 Client-Side Adaptive Throttling#

# per client, per backend, over a trailing 2-minute window
p_reject = max(0, (requests - K * accepts) / (requests + 1))   # K = 2
if random() < p_reject: fail locally (counts as a request, not an accept)

No states, no cliff. At K=2 the client sends at most ~2× what the backend is accepting, so the backend still sees enough traffic to signal recovery. Lower K is more aggressive; higher K wastes more backend rejection work.

A.3 Adaptive Concurrency (AIMD sketch)#

limit = 20
on response(rtt):
  if rtt <= rtt_min * 1.5 and inflight >= limit * 0.9: limit += 1       # additive increase
  if rtt >  rtt_min * 2.0 or dropped:                   limit = max(5, limit * 0.9)  # multiplicative decrease
admit if inflight < limit else reject 503

Gradient-based variants (as in Netflix's concurrency-limits) compare long-term vs short-term RTT and converge faster; all share the idea that latency growth means queueing, and queueing means you are past capacity.

A.4 Bulkhead Variants#

Appendix B: Criticality and Dependency Classification#

Appendix C: Coordination Mechanisms — Where Protection Lives#

C.1 Quick Comparison#

See Rate Limiting for the policy side.

Appendix D: API Contract and Client Behavior#

D.1 Deadline Propagation#

Use absolute deadlines on the wire (gRPC converts to a relative timeout per hop and back); relative "timeout" headers accumulate skew and ignore queueing time.

D.2 Rejection Responses#

D.3 Retry Thundering Herd#

Exponential backoff without jitter synchronizes clients: all retry at 100ms, 200ms, 400ms. Full jitter (sleep = random(0, min(cap, base × 2^n))) spreads them uniformly; the AWS Builders' Library analysis shows full jitter completes work with markedly fewer total calls than un-jittered backoff under contention.

Appendix E: Observability#

E.1 Core Metrics — Non-Negotiable#

# Caller side
breaker_state{caller, dependency}                        gauge
breaker_transitions_total{caller, dependency, to}        counter
bulkhead_active{caller, dependency} / bulkhead_max       gauge
fallback_served_total{caller, dependency, kind}          counter
fallback_last_served_timestamp{dependency}               gauge
deadline_exceeded_total{caller, dependency}              counter

# Mesh
rpc_retry_ratio{src, dst}                                gauge (retries / first attempts)
outlier_ejections_active{cluster}                        gauge

# Callee side
shed_total{service, criticality, reason}                 counter
expired_on_arrival_total{service}                        counter
goodput_rps{service} vs throughput_rps{service}          gauge
queue_wait_ms_p99{service}                               histogram

E.2 Critical Alerts#

E.3 Control Plane vs Data Plane#

The data plane (proxies, in-process breakers) must keep working with the last-known-good config when the control plane is down. A mesh that fails closed when its control plane is unreachable turns a platform outage into a fleet outage. Test it: kill the control plane in staging monthly.

E.4 Debugging "Everything Is Up but Nothing Works"#

1. Compare goodput to throughput at each tier — find where work is being done and discarded.
2. Check rpc_retry_ratio along the call graph — find the amplifier.
3. Check expired_on_arrival_total — a high value means queues are full of dead work.
4. Check breaker transitions for periodicity — a sawtooth means synchronized breakers.

Appendix F: Scale Evolution#

F.1 What Works at Each Scale#

F.2 Multi-Region Path#

Regional isolation → edge-level whole-user failover → cells within regions → shuffle sharding for multi-tenant services. Per-dependency cross-region failover stays a narrow, capped exception.

F.3 What You Don't Build on Day One#

• A shared, distributed breaker state store
• Custom adaptive-concurrency algorithms (use a library)
• Per-method breaker tuning for every endpoint (start per dependency)
• Automated regional evacuation (start with a tested manual runbook)
• Hedged requests (add only for read paths with strict tail SLOs, with a budget)

Appendix G: Multi-Tenancy, Fairness and Cost#

G.1 Noisy Neighbor via Retries#

One tenant's malformed requests fail, their SDK retries aggressively, and the shared service sheds — hitting every tenant. Fixes: per-tenant concurrency quotas at the callee; shed by tenant and criticality; per-tenant retry budgets at the gateway.

G.2 Shuffle Sharding#

Assign each tenant to a random subset of k hosts out of n (e.g., 2 of 16). A poison tenant takes down only its 2 hosts; the probability another tenant shares both is 1/C(16,2) = 1/120. Bounded blast radius without a cell per tenant. Amazon has written publicly about using this for Route 53.

G.3 Cost of Resilience Features#

G.4 Tradeoff Summary#