Degraded Mode Framework — Cross-Cutting Pattern

Technologies that implement this pattern: API Gateways · Redis · Apache Kafka · ZooKeeper & etcd · DynamoDB

Why This Matters#

Every system degrades. The only choice you get is whether it degrades in the order you chose in advance or the order the failure chooses for you at 3am. Uncontrolled degradation is what an outage is: the recommendation service slows down, holds threads, exhausts the pool that checkout also uses, and a non-essential widget takes the revenue path with it.

Most candidates treat degraded mode as a resilience-technique question — circuit breakers, retries, timeouts. Staff engineers treat it as a product-priority question decided before the incident. Which features are critical, which can get worse, which can vanish? What does the user see in each tier? Who is allowed to push the system down a tier, and does that decision need a human? Circuit breakers are the mechanism; the degradation ladder — signed off by product, wired to automated triggers, and rehearsed — is the design.

The second reframe: a fallback you haven't exercised is a second outage waiting to happen. Fallback code runs rarely, so it rots: it calls the same failed dependency, it was never load-tested, the static snapshot is six months old. Degraded modes are only real if they run in production on purpose — brownouts, game days, a small percentage of traffic permanently on the fallback path.

If you can present a tiered ladder with entry/exit criteria, map each feature to a criticality class, say who decides each transition, and explain how you know the fallbacks work, you are answering at Staff level.

The 60-Second Version#

• Classify every feature into three buckets: Critical, Degradable, Sheddable. In a typical product page, ~10–20% of calls are Critical (auth, price, add-to-cart), ~30% Degradable (search, recommendations with a generic fallback), ~50% Sheddable (counts, badges, "people also viewed"). Shedding that 50% frees roughly half the fan-out capacity.
• Define 4–5 tiers with numeric entry and exit criteria. E.g., enter Tier 1 when p99 > 2× SLO for 2 minutes or CPU > 80%; exit only after 10 minutes below threshold. Exit hysteresis of 3–5× the entry window prevents flapping.
• Automate the first two tiers; require a human for the last two. Shedding sheddable features should happen in <30 seconds without a page. Going read-only or to a static site is a business decision — the incident commander owns it under a pre-signed policy.
• Static fallbacks must be precomputed and dependency-free. A "last known good" snapshot on object storage/CDN, refreshed every 5–15 minutes, served with zero calls to the failing tier. If the fallback path needs the database, it isn't a fallback.
• Brownouts test and relieve at the same time. Disable optional components for a dynamic fraction of requests (0–100%) under load, or deliberately for 1–5% of traffic in steady state, so the fallback path is always warm and always proven.
• Kill switches must not depend on the thing that's failing. The flag/config system needs its own availability ≥ the services it controls, a local cached value on every host, and a break-glass path that works when the control plane is down.

The Problem#

Demand exceeds capacity, or a dependency fails, and the system has to keep doing something useful. Without a plan, degradation is decided by thread pools and timeouts: whichever dependency is slowest consumes the shared resources and every feature fails together. A ride-hailing app that can't compute surge pricing should still dispatch at the last known multiplier. A streaming service whose personalization is down should still play video with a generic row of popular titles. A checkout whose fraud model is timing out needs someone to have decided — in daylight, with finance in the room — whether to approve small orders or block them all. The engineering problem is building those modes; the Staff problem is making sure the decisions are made before anyone needs them.

Case Studies That Use This Pattern#

• Circuit Breakers — The mechanism that trips a dependency into its fallback; degraded mode is what the fallback does
• Rate Limiting — Fail-open vs fail-closed when the limiter's own state store is down
• Payment Processing — Fraud/risk dependency down: approve-below-threshold vs block decisions owned by finance
• News Feed — Ranking down → chronological feed; the canonical degradable feature
• Flash Sales & Ticketing — Waiting rooms as a planned Tier 4 under demand spikes
• API Gateway — Where criticality headers are enforced and load is shed first
• Notification System — Deferring non-urgent notifications while keeping security alerts flowing

The Four Intents#

"Add graceful degradation" can mean four different problems. They need different mechanisms and different owners.

🎯 Staff Move: "I'll design for overload and dependency failure first, since that's where 80% of incidents come from. But I want to flag up front that auth and payments fail closed — or fail open only under a limit finance has signed — and that's not an engineering call I'll make alone."

The Degradation Ladder#

The artifact at the center of this pattern: a small number of tiers, each with a user-visible description, entry and exit criteria, and a decision owner.

🎯 Staff Insight: Recovery is the most dangerous part of the ladder. Re-enabling all features at once is a self-inflicted thundering herd — cold caches, reconnecting clients, and queued writes all hit together. Step up one tier at a time, with a dwell time, and ramp traffic within a tier (10% → 50% → 100%).

The Core Tradeoff#

Staff Default Position#

Decide the degradation order in daylight, automate the reversible tiers, and prove every fallback in production.

Classify features into Critical / Degradable / Sheddable with product sign-off. Build a 4–5 tier ladder with numeric entry and exit criteria. Automate Tier 1–2 transitions (reversible, low business risk) with hysteresis; require the incident commander for Tier 3+ under a pre-approved policy so nobody is waiting on a VP at 3am. Make fallbacks dependency-free (precomputed snapshots, cached config, local defaults). Run the fallback path continuously for a small slice of traffic or on a regular brownout schedule so it cannot silently rot.

When to Deviate#

• Correctness-critical flows — Auth, payments, permissions, medical or safety systems. Default to fail-closed; any fail-open needs explicit limits (e.g., approve orders < $50 for ≤15 minutes) signed by the owning business function.
• Single-purpose systems — A system with one feature (a payment ledger, a DNS resolver) has nothing to shed. Invest in redundancy and static stability instead of a ladder.
• Early-stage products — Under ~10 services and one team, a full ladder is overhead. Start with timeouts, one kill switch per non-critical dependency, and a static maintenance page.
• Regulated freshness — Where showing stale data is itself a violation (trading prices, drug dosage), a stale fallback is worse than an error. Degrade to "unavailable" with a clear message.

The L5 → L6 → L7 Contrast#

The Five Fault Lines#

Fault Line 1: Fail-Open vs Fail-Closed#

When a protective dependency (rate limiter, fraud model, authz service) fails, you either let traffic through or block it. Who pays: fail-open pays in exposure (fraud, abuse, data access); fail-closed pays in availability for legitimate users. Staff default: fail-open for abuse protection and rate limiting (the harm of blocking everyone exceeds the harm of 15 minutes without limits), fail-closed for authorization and money movement, with bounded fail-open (amount caps, time caps, known-good users only) where the business signs it. Deviate when: the protective system is itself the product (a WAF vendor, a compliance gate).

Fault Line 2: Automated vs Human Triggers#

An automated trigger reacts in 10–30 seconds; a paged human takes 5–15 minutes to acknowledge, orient, and act. Who pays: automation pays in false positives (shedding recommendations because of a noisy metric); humans pay in minutes of full outage. Staff default: automate reversible, low-business-impact tiers; human-in-the-loop for tiers that stop revenue or risk data. Every automated trigger has hysteresis and a rate limit on transitions (e.g., max 1 tier change per 5 minutes). Deviate when: the event is predictable (a ticket drop) — pre-enter Tier 1 on a schedule.

Fault Line 3: Precomputed vs Live Degraded Fallbacks#

A live fallback computes a simpler answer (popular items from a lighter query); a precomputed fallback serves a snapshot from object storage. Who pays: live fallbacks still consume capacity on the degraded tier — sometimes the same capacity that's exhausted; precomputed fallbacks cost a snapshot pipeline and staleness. Staff default: precomputed for Tier 2 (must survive total backend loss), live for Tier 1 (still have most capacity). Deviate when: the content is per-user and can't be meaningfully precomputed — degrade to a generic version instead.

Fault Line 4: Server-Driven vs Client-Driven Degradation#

Servers can flag degradation in responses; clients can also degrade themselves (cached last response, hide widgets on timeout). Who pays: server-driven depends on reaching the server; client-driven means old app versions behave differently and mobile teams own part of the reliability story. Staff default: both — the server sends a degradation hint header, and clients have a local timeout budget per component and render cached content when the call fails. Deviate when: clients are third parties you can't update (public APIs) — degrade only server-side and document it in the API contract.

Fault Line 5: Practice in Production vs Hope#

Brownouts and game days impose small, deliberate user impact. Who pays: product accepts a measurable dip (typically <0.1% of sessions affected) in exchange for knowing the fallback works. Staff default: continuous low-percentage fallback traffic for Tier 1–2; quarterly game days for Tier 3–4. Deviate when: a regulated flow can't be degraded deliberately — test in a production-shaped staging environment with production traffic replay.

Common Interview Mistakes#

Quick Reference#

Staff Sentence Templates#

"Before I add breakers, I want the degradation order signed off: [features A, B] are sheddable, [C] degrades to [a precomputed fallback refreshed every N minutes], and [D, E] are critical. Product owns that list; I own making each tier actually work."

"Tiers 1 and 2 are automatic because they're reversible and cost only [engagement metric]. Tier 3 stops [revenue flow], so the incident commander decides under a policy [business owner] pre-approved — nobody waits for a VP at 3am."

"The fallback for [feature] must not share [thread pool / database / config service / region] with the primary path, otherwise it fails in exactly the situation it exists for."

"For [protective control], we fail [open/closed]. If open, exposure is capped at [amount/time] and [team] is paged the moment we enter that mode."

Implementation Deep Dive#

1. Criticality Propagation + Priority Load Shedding#

Google's SRE book describes tagging every request with a criticality (e.g., CRITICAL_PLUS, CRITICAL, SHEDDABLE_PLUS, SHEDDABLE) that propagates through the call graph, so an overloaded backend rejects sheddable work first. The same idea works with a single header.

# At the edge (API gateway): classify once, propagate everywhere
CRITICALITY = {
    "POST /checkout":        3,   # critical
    "GET  /product/:id":     2,   # degradable (core content)
    "GET  /recommendations": 1,   # sheddable_plus
    "POST /analytics/beacon":0,   # sheddable
}
function edge(req):
    req.headers["x-criticality"] = CRITICALITY.get(route(req), 1)
    req.headers["x-deadline-ms"] = now_ms() + 800                   # end-to-end budget

# In every service: admission control before doing work
function admit(req):
    c = int(req.headers["x-criticality"])
    util = concurrency_in_use / concurrency_limit                   # adaptive limit
    threshold = { 0: 0.60, 1: 0.75, 2: 0.90, 3: 1.00 }[c]           # shed lowest first
    if util > threshold:
        metrics.incr("shed.rejected", tags=["crit:" + c])
        return reject(503, retry_after = 2 + rand(0, 3))           # jittered, bounded
    if now_ms() > int(req.headers["x-deadline-ms"]):
        metrics.incr("shed.expired")
        return reject(504)                                          # don't work on dead requests
    return process(req)

Why it matters: without tags, an overloaded service rejects uniformly — 30% of checkouts fail alongside 30% of analytics beacons. With tags, analytics goes to 100% rejection before a single checkout fails. Downstream calls inherit the caller's criticality; a batch job can't escalate itself to critical.

🎯 Staff Insight: The hard part isn't the admission code — it's getting 40 services to tag honestly. If every team marks its traffic critical, the scheme collapses. Criticality is assigned at the edge by route, reviewed by the reliability team, and can only be lowered downstream.

2. The Ladder Controller — Automated Tiers with Hysteresis#

TIERS = [
  { tier: 1, enter: "p99_ms > 2*SLO for 120s OR cpu > 0.80 for 120s", exit_dwell: 600 },
  { tier: 2, enter: "crit_error_rate > 0.01 for 60s OR breaker_open(core_dep)", exit_dwell: 900 },
]
MAX_TRANSITIONS_PER_5MIN = 1

every 10s:
    signals = read_metrics()
    target = highest tier whose enter condition holds         # auto tiers only (1-2)
    if target > current and transitions_recent() < MAX_TRANSITIONS_PER_5MIN:
        set_tier(target)                                      # step down immediately
        page(IC, "entered tier " + target, signals)
    elif target < current and healthy_for(current.exit_dwell):
        set_tier(current - 1)                                 # step up ONE tier at a time
        ramp_features(current, steps=[0.1, 0.5, 1.0], interval=120)

function set_tier(t):
    flags.publish("degrade.tier", t)                          # pushed to all hosts
    audit_log.write(actor="ladder-controller", tier=t, signals=signals)

Hosts read the tier locally: every host caches degrade.tier in memory with a last-known-good value persisted to local disk. If the flag service is unreachable, hosts keep the last value — they do not reset to Tier 0.

3. Dependency-Free Static Fallback#

# Snapshot job: every 10 minutes, render fallback content from healthy state
every 10m:
    for surface in ["home", "category/*", "product/top-50K"]:
        html_or_json = render(surface, personalization=false)
        s3.put("fallback/" + surface, html_or_json, cache_control="max-age=900")
    s3.put("fallback/_manifest", { generated_at: now(), count: n })
    metrics.gauge("fallback.snapshot_age_s", 0)

# Serving path: consult tier BEFORE calling any backend
function serve(req):
    if local_tier() >= 2 or breaker("catalog").open:
        body = local_disk_cache.get(req.surface) or cdn_fetch("fallback/" + req.surface)
        return respond(body, headers={"x-degraded": "tier2", "Cache-Control": "max-age=60"})
    return live_render(req)

Rules: the snapshot is stored where the failing tier isn't (object storage + CDN), refreshed often enough that staleness is product-approved (here 10 min, max 25 min with a missed run), and alerts when fallback.snapshot_age_s > 1,800. The serving path checks the tier first, so in Tier 2 not even a timed-out call is made.

4. Brownout Dimmer#

The brownout idea (Klein et al., ICSE 2014) is a "dimmer" in [0, 1] that sets the probability of executing optional content per request, adjusted continuously from a latency target.

dimmer = 1.0                       # 1.0 = all optional features on
TARGET_P95_MS = 300

every 5s:
    err = (TARGET_P95_MS - p95_ms()) / TARGET_P95_MS
    dimmer = clamp(dimmer + 0.3 * err, 0.02, 1.0)   # floor 2%: fallback always exercised
    metrics.gauge("brownout.dimmer", dimmer)

function render(req):
    page = core(req)
    if rand() < dimmer: page.recs = recommendations(req)  else: page.recs = popular_fallback()
    if rand() < dimmer: page.counts = counts(req)
    return page

Why the 2% floor: the fallback path runs for at least 2% of traffic all the time, so it is load-tested and monitored continuously — a broken fallback surfaces as an alert on a Tuesday afternoon, not during a Sev-1.

5. Read-Only Mode with Deferred Writes#

function handleWrite(req):
    if local_tier() >= 3:
        if req.kind in DEFERRABLE:              # reviews, profile edits, uploads metadata
            queue.put(req, idempotency_key=req.key)   # durable queue outside failing DB
            return 202, {"status": "queued", "message": "Saved; will appear shortly"}
        return 503, {"message": "Temporarily read-only", "retry_after": 300}
    return db.write(req)

Replay on recovery goes through the same idempotency keys and rate-limits itself to ~20% of primary write capacity so recovery doesn't become the next overload.

Mechanism Comparison

Architecture Diagram#

How to narrate it: the data plane never asks the control plane for permission at request time. Tiers and dimmers are pushed and cached locally; fallback assets live outside the failing tier; checkout has its own pool so a slow recommendations call can't consume its threads.

Failure Scenarios#

1. The Fallback That Called the Outage — 100% Error Rate From a 5% Feature#

Personalization's fallback was "fetch the popular list from the catalog DB." When the catalog replica fleet slowed down, both the primary path and the fallback hit the same replicas.

t=0      Catalog replicas p99 40ms -> 900ms (bad query plan after stats refresh).
t=+20s   Personalization times out; breaker opens; fallback fires.
t=+25s   Fallback queries the same catalog replicas: load +60%.
t=+60s   Page service thread pool (shared) saturated waiting on fallback.
t=+90s   Home page error rate 100%, including checkout links rendered there.
t=+9min  On-call disables personalization AND fallback via flag; page renders.
t=+40min Query plan fixed; features restored one at a time.

Detection: fallback.invocations spike correlated with dep.latency_p99{dep=catalog}; threadpool.saturation{pool=page} > 90%. Blast radius: entire home page — a sheddable feature took down a critical surface. Mitigation: fallback moved to precomputed snapshot in object storage; separate bulkhead pool per dependency. Prevention: design review checklist: "What does the fallback depend on? Is any of it shared with the primary path?"; 2% brownout floor so the fallback runs under real load daily. Owner: personalization team owns the fallback; reliability team owns the checklist.

🎯 Staff Insight: The rule is shared fate audit: a fallback must differ from the primary path in every resource that can fail — dependency, pool, region, config. If you can't name the difference, you don't have a fallback.

2. Ladder Flapping — Tier Oscillation Every 90 Seconds#

An automated ladder entered Tier 1 at CPU > 80% and exited at CPU < 80%, with no dwell. Shedding dropped CPU to 65%, which exited Tier 1, which restored load, which re-entered Tier 1.

t=0      CPU 82% -> Tier 1 (recs, counts shed). CPU drops to 65%.
t=+30s   Exit condition met -> Tier 0. Cold recs caches refill, CPU 95%.
t=+60s   Tier 1 again. Users see widgets appear and vanish.
t=+15min 20 transitions. Recs cache never warms; CDN hit ratio falls on page variants.
t=+17min On-call pins Tier 1 manually. Stable.

Detection: degrade.tier_transitions > 3 per 10 min. Blast radius: all users on affected surfaces; cache churn raised backend load 25% above the no-ladder baseline. Mitigation: manual pin; exit dwell of 10 min; max 1 transition per 5 min; exit threshold 20 points below entry (80% in, 60% out). Owner: reliability team owns the controller and its parameters.

3. The Kill Switch That Needed the Outage to Work#

A regional networking event made the flag service unreachable from half the fleet. On-call flipped the kill switch for a failing payment-risk dependency; only hosts that could reach the flag service received it. Worse, hosts configured to "fetch flags on startup" were being restarted by auto-healing and came up with defaults — Tier 0, risk checks enabled, calling the dead dependency.

t=0      Risk service unavailable in region A. Checkout p99 3s, errors 35%.
t=+4min  IC approves pre-signed policy: bypass risk for orders < $50 for 15 min.
t=+5min  Flag flipped. 48% of hosts receive it. Error rate falls to 18%.
t=+8min  Auto-healing restarts unhealthy hosts; they boot with defaults (risk on).
t=+20min Break-glass: env override pushed via deployment pipeline. Errors 0.5%.
t=+35min Risk recovers; override removed; exposure review: $41K in un-scored orders.

Detection: flags.fetch_failures per host; flags.version_skew (hosts on different flag versions) > 5%. Blast radius: half of checkout for 20 minutes. Mitigation: break-glass override via deploy pipeline; bypass bounded by amount and time. Prevention: flags persisted to local disk and loaded before startup defaults; flag service deployed per region with static stability; break-glass path documented and drilled. Owner: platform team owns the flag system's availability SLO (set higher than any service it controls); payments owns the bounded-bypass policy with finance.

Operational Reality Matrix#

Who Decides — The Decision Rights Matrix#

Degraded mode fails organizationally before it fails technically: on-call knows how to flip the switch but doesn't know whether they're allowed to turn off checkout for 5% of users. Decide this in advance and write it down.

🎯 Staff Move: "The key design output isn't the flag system — it's this table, signed before launch. At 3am the incident commander should be executing a decision, not making one."

The Principal Lens#

Why L7 Sees This Problem Differently#

A Staff engineer designs a ladder for their system. A Principal engineer sees that the company's real failure posture is the composition of fifty ladders — and that the composition has properties no single team designed. Tier 2 on the product page may be fine; Tier 2 on product page + search + cart at the same time may leave users unable to buy anything. Every team's kill switch lives in the same flag service; every fallback snapshot lives in the same bucket in the same region. At org scale, degraded mode becomes a question of correlated failure and shared decision rights: one vocabulary of tiers, one control plane engineered to outlive everything it controls, and a practice budget (game days, brownouts) that the org actually spends.

The Org-Level Fault Line#

A central degradation control plane with org-wide tiers vs per-team fallbacks.

Meta's published Defcon system (OSDI 2023) is a public example of the third shape: feature "knobs" registered by product teams, grouped by business criticality into levels, and actuated centrally during capacity crises.

Cost Model#

Assumptions: fully loaded engineer ~$25K/month; snapshot storage and CDN at object-storage pricing; game-day cost measured in engineer-hours; revenue at risk estimated as hourly revenue × outage minutes avoided.

The Principal framing: at $5B ARR, revenue is ~$570K/hour. If the degradation program turns two 60-minute full outages per year into 60-minute Tier 2 events that retain ~80% of revenue, it preserves roughly $900K/year in direct revenue — before counting trust and SLA credits. That's the number that funds the platform team.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Degradation Tiers and Fallback Standard (v1)

Scope: All user-facing services and any service with a Critical-tagged caller.

MUST:

1. Each service declares its features/endpoints as Critical, Degradable, or Sheddable, approved by the owning product lead and reviewed annually.
2. Each service defines its behavior at org tiers T1–T4; "no change" is allowed but must be explicit.
3. Fallbacks for Degradable features MUST NOT share a dependency, thread pool, or region with the primary path; the design doc names the difference.
4. Degradation state MUST be read from local cache with last-known-good persistence; no request-path calls to the control plane.
5. Requests MUST carry an edge-assigned criticality; services may lower but never raise it.
6. Fail-open of authorization is prohibited. Fail-open of financial controls requires a signed cap (amount, duration) and auto-expiry.

SHOULD: Run fallback paths for ≥1% of production traffic continuously; participate in a quarterly game day.

Exceptions: Reliability review, time-boxed to 2 quarters, owner named.

Success metrics: % of Critical-path dependencies with tested fallbacks (target > 90%); minutes at Tier 2+ vs minutes of full outage per quarter; zero incidents where a fallback shared fate with its primary; median time-to-degrade < 60s.

What I'd Tell the VP#

When something breaks today, the whole site tends to break with it, because nothing decides what's allowed to fail first. I want us to agree — product and engineering together, ahead of time — on which features are essential, which can get simpler, and which can switch off, and then automate the first steps so we degrade in seconds instead of having an outage for minutes. The cost is a small platform team and about a day a quarter from each product team to practice. The payoff is that our worst incidents become "the site was plainer for an hour" instead of "the site was down for an hour." I need product leaders to sign off on the essentials list and finance to set the limits for the payment exceptions.

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "I'd add a fallback and a kill switch for each dependency." — Right per service, silent on shared fate across services and on who's allowed to use the switch.
• "We'll automate the ladder so humans aren't needed." — Ignores that the expensive tiers are business decisions that need pre-delegated authority, not automation.
• "We run chaos tests in staging." — Doesn't commit the org to practicing in production where fallbacks actually rot.

In the Wild#

Google: Request Criticality and Load Shedding#

The Google SRE book's chapters on handling overload describe tagging requests with a criticality level (from CRITICAL_PLUS down to SHEDDABLE) that propagates across RPCs, so that overloaded backends reject the least important work first. It also describes client-side adaptive throttling, where clients locally reject a fraction of requests when their backend has been rejecting them, so retries don't amplify overload.

Staff insight: The mechanism that keeps critical traffic alive is not a clever algorithm; it's an org-wide convention (criticality propagated in every call) that every service honors. That's the argument for making degradation a shared vocabulary rather than a per-team feature.

AWS: Static Stability#

The Amazon Builders' Library describes "static stability": systems designed so that the data plane keeps working without the control plane — for example, pre-provisioning enough capacity across Availability Zones that losing one AZ doesn't require launching new instances during the event. The data plane keeps serving with what it already has.

Staff insight: This is the design principle behind "the kill switch must not depend on what's failing." Degraded modes that require provisioning, deploying, or reaching a control plane during the incident are fallbacks in name only.

Meta: Defcon — Graceful Feature Degradation#

Meta's OSDI 2023 paper on Defcon describes a system where product teams register "knobs" that turn off or reduce features, grouped into levels by business criticality, which can be actuated during capacity crises (such as losing a datacenter's worth of capacity) to cut load while keeping core functionality. The paper reports that turning off lower-priority features freed substantial capacity during real events, and that knobs are tested regularly.

Staff insight: Defcon is the "federated ladder with central actuation" shape: teams own what their knobs do, the org owns the levels and the button. It's the strongest public citation for the Principal answer.

Practice Drill#

Prompt: "Our e-commerce home page calls 14 services. Last month, the reviews service had a memory leak and the whole home page went down for 35 minutes, including links to checkout. Design degraded mode so this can't happen again."

Staff Interview Application#

How to Introduce This Pattern#

"Before I add circuit breakers, I want to decide the order in which this system is allowed to get worse. I'll split features into critical, degradable, and sheddable with product, define three or four tiers with numeric entry and exit criteria, automate the reversible ones, and name who's allowed to trigger the rest. Then I'll make sure every fallback runs in production a little bit every day, so we know it works."

When NOT to Use This Pattern#

• Nothing to shed: Single-purpose systems (a ledger, a lock service) should invest in redundancy and static stability, not ladders.
• Stale is worse than nothing: Regulated or safety data where a stale fallback misleads. Degrade to an explicit "unavailable."
• Tiny scale: Under ~10 services, timeouts plus a handful of kill switches and a maintenance page deliver most of the value; a ladder controller is overhead.
• Batch/offline systems: Pipelines degrade by delaying, not by shedding features — use backpressure and SLAs on lateness instead (Data Pipeline Patterns).

Follow-Up Questions to Anticipate#

Evaluation Rubric#

Strong Hire Signals

Lean No-Hire Signals

Common False Positives: Knowing Hystrix/resilience4j configuration ≠ designing degraded modes. A chaos-engineering vocabulary ≠ having exercised fallbacks. "We have feature flags" ≠ having a degradation ladder.

Capacity Planning Quick Reference#

Sizing the Degraded Paths#

freed_capacity(tier)   = Σ fanout_cost(features shed at tier) / total_fanout_cost
required_tier(load)    = lowest tier where capacity × (1 + freed_capacity) ≥ load
snapshot_staleness_max = refresh_interval + job_runtime + one missed run
bulkhead_slots(dep)    = dep_qps × dep_timeout_s × 1.5
recovery_ramp          = 10% → 50% → 100% over ≥ 6 min per tier

Key Numbers Worth Memorizing#

Common Pitfalls Checklist#

• Every feature has a criticality class signed by product
• Every tier has numeric entry/exit criteria and hysteresis
• Fallbacks share no dependency, pool, or region with the primary path
• Degradation state is cached locally with last-known-good persistence
• Recovery steps up one tier at a time with a traffic ramp
• Fail-open decisions have signed caps and auto-expiry; authz never fails open
• Fallback paths run in production continuously, and game days happen on a calendar
• The decision-rights table exists and on-call has read it