Technologies referenced in this case study: Kafka · Redis · PostgreSQL · DynamoDB · Time Series DBs · API Gateways

Related case studies: Load Balancer · Circuit Breakers · Rate Limiting · Flash Sales · Metrics & Monitoring · Degraded Mode · Back-of-Envelope Estimation

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once. Return to individual sections for targeted review.

Reactive-only autoscaling, CPU target 70%, no pre-warming:
t=0:       Launch email to 40M users. RPS climbs from 20K to 140K in 90 seconds.
t=+30s:    CPU crosses 70%. Autoscaler (60s evaluation window) hasn't fired yet.
t=+75s:    Scale-out requested: +80 instances.
t=+90s:    p99 latency 4s. Clients time out and retry, so effective RPS is 210K.
t=+3min:   New instances boot. JVM cold, caches cold. They take full traffic and fail health checks.
t=+4min:   LB ejects "unhealthy" new instances; autoscaler replaces them. Death spiral.
t=+6min:   New app instances open 6,400 DB connections. Postgres max_connections 5,000 → DB refuses.
t=+25min:  Manual intervention: shed load, cap scale-out, raise pool limits. Launch is a headline.

Staff design (scheduled pre-scale + reactive on concurrency + protected dependencies):
t=-60min:  Scheduled action raises min capacity to forecast peak × 1.3 (load-tested last week).
t=-30min:  Warm pools filled; caches primed by synthetic traffic.
t=0:       Same 140K RPS. Utilization peaks at 58%. p99 stays at 180ms.
t=+10min:  Actual traffic exceeds forecast by 20%. Reactive policy on in-flight requests adds 15%.
t=+12min:  Admission control at the gateway sheds 2% of low-priority traffic during the step.
t=+3h:     Scale-in, gradually, over 45 minutes. Cost of pre-scale: ~$4K. Cost of the alternative: the launch.

Executive Summary

If you only read one section, read this. Everything else in the case study elaborates on the contrast below.

What This Interview Actually Tests#

Auto-scaling is not a controller question. The formula desired = current × (metric / target) fits on one line.

This is a risk pre-payment question that tests:

• Whether you understand that a control loop with a 3–5 minute actuation delay cannot react to a 90-second spike, and that someone must pay for capacity before demand arrives
• Whether you pick a scaling signal that measures demand, not a side effect of demand
• Whether you find the components that can't scale (databases, connection limits, quotas, third parties) and protect them from the parts that can
• Whether you can price headroom in dollars and name who pays for idle capacity versus who pays for the outage

The key insight: every capacity decision trades idle dollars for incident risk. Autoscaling narrows the gap between them without closing it. Staff engineers state the headroom they are buying, the scenario it covers, and who signed off on the scenarios it doesn't.

The L5 → L6 → L7 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll design for reliability through known peaks with a survival floor. That means a user-facing stateless tier with a strong diurnal curve plus planned launches, and enough standing headroom to lose one availability zone without paging anyone. Pure cost efficiency is a tuning exercise on top of that design, and the reverse doesn't hold."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: citing well-known production practice shows you've studied operational reality. Use these as one-sentence anchors in the interview.

Netflix — Predictive Scaling and Regional Evacuation#

Netflix publicly described Scryer, a predictive autoscaling engine that forecasts demand from historical patterns and provisions capacity ahead of the diurnal ramp, because reactive scaling lagged their steep evening rise. Separately, Netflix's regional failover exercises (Chaos Kong) evacuate a whole AWS region and shift its traffic to the others, which requires each remaining region to absorb a large traffic increase within minutes.

Staff insight: two ideas worth citing. Predictive scaling exists because boot time exceeds ramp time. Region evacuation means your headroom target is set by the failover scenario, not by normal peak. Say both.

Facebook — Kraken: Load Testing With Live Traffic#

Facebook's Kraken (published at OSDI 2016) shifts live production traffic toward a subset of servers, clusters or regions to measure their true capacity limits, guided by health metrics that stop the test before users are hurt.

Staff insight: synthetic load tests miss real request mixes, cache hit rates and dependency behavior. Kraken's lesson is that the most accurate capacity number comes from production, under controlled conditions. Mention it when asked "how do you know your capacity?"

Kubernetes & AWS — Controllers With Built-In Asymmetry#

The Kubernetes Horizontal Pod Autoscaler evaluates metrics every 15 seconds by default and applies a default 5-minute scale-down stabilization window, so it scales out quickly and in cautiously. AWS Auto Scaling offers target tracking, scheduled actions, warm pools and a predictive scaling mode driven by historical load. Karpenter and the Cluster Autoscaler add nodes when pods can't be scheduled.

Staff insight: the defaults encode the Staff position. Scale out fast because under-provisioning is expensive. Scale in slowly because flapping is expensive and idle is cheap. Being able to quote these defaults shows you've operated these systems.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: the controller has three inputs. It measures demand (reactive), takes a floor from forecasts and the event calendar (predictive), and takes a ceiling from the capacity model's dependency budgets. The gateway's admission control is the safety valve for the gap between demand and capacity. The warm pool shortens the actuation delay from minutes to ~30 seconds. The database sits outside the elastic tier on purpose: it scales in hours, and the design protects it rather than pretending it's elastic.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: candidates draw an autoscaling group, write "CPU > 70% → add instance," and stop there. Everything that decides whether the launch survives comes after that sentence: the actuation delay, the dependency that doesn't scale, the load test that lied, and who paid for the headroom.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional goal in one sentence:

"We want the service to meet its latency SLO as demand changes, without paying for peak capacity 24/7."

Then frame the real requirements:

"Three intents pull in different directions: minimizing cost on a variable curve, surviving known peaks like launches, and surviving unknown spikes and failures like a zone outage. I'll design for known peaks with a survival floor. My constraints are p99 under 300ms, at most 1% of requests shed at the worst minute, the ability to lose one availability zone without paging, and a cost target of no more than ~40% idle averaged over a day."

Establish the shape of demand, because it determines everything:

"What does the curve look like? I'll assume a diurnal pattern from 20K RPS at night to 100K at peak, rising over about 2 hours, plus planned launches that step from 100K to 250K in under 2 minutes. The diurnal curve is slow enough for reactive scaling. The launch step isn't."

🎯 Staff Move: Asking for the rise time of demand, not just the peak, is the question that separates levels. Compare it to your provisioning time and you know which tool you need before drawing anything.

Phase 2: Core Entities & API (1–2 minutes)#

The nouns:

• Capacity unit — the measured throughput of one instance at SLO (e.g., 400 RPS per 4-vCPU instance at p99 < 300ms). The single most important number in the design.
• Scaling policy — signal, target, min, max, cooldowns, step limits.
• Capacity floor — scheduled or predicted minimum for a time window.
• Dependency budget — max connections/RPS each downstream grants this service.
• Event — a calendar entry with expected multiplier, owner, and pre-scale window.

The control API (platform-facing, not end-user):

PUT  /services/{svc}/scaling-policy
     { signal: "inflight_per_instance", target: 60, min: 40, max: 300,
       scale_out: { max_step_pct: 50, cooldown_s: 60 },
       scale_in:  { max_step_pct: 10, stabilization_s: 600 } }

POST /services/{svc}/capacity-floors
     { start: "2026-11-27T13:00Z", end: "...", min_instances: 220, reason: "BF launch", approver: "..." }

GET  /services/{svc}/capacity-model
     → { rps_per_instance_at_slo: 400, max_safe_instances: 280,
         limiting_dependency: "orders-db connections", headroom_pct: 38 }

🎯 Staff Move: "The capacity model endpoint matters more than the policy. If a team can't tell me their RPS per instance at SLO and their limiting dependency, their autoscaling policy is a guess. Everything else I design assumes that number exists and is re-measured regularly."

Phase 3: High-Level Architecture (≤5 minutes)#

Walk it in 90 seconds:

1. The service emits in-flight requests per instance every 15 seconds.
2. The reactive controller computes desired = ceil(current × observed / target) and clamps it between the floor and the ceiling.
3. The floor comes from the forecast (trailing-weeks seasonality) and the event calendar (pre-scales for launches).
4. The ceiling comes from the capacity model: the smallest of budget, DB connection budget ÷ per-instance pool size, and third-party quota ÷ per-instance call rate.
5. New capacity comes from the warm pool (~30s) before cold boot (~3 min). The LB slow-starts new targets over 60 seconds.
6. If demand exceeds what the fleet can serve, the gateway sheds lowest-priority traffic first. It returns 503 with Retry-After rather than letting latency collapse for everyone.

🎯 Staff Move: "That's the design most people draw, minus the floor and ceiling. The floor handles what we know is coming. The ceiling stops us from DDoSing our own database. Admission control handles what neither covers. Now let me show where it breaks."

Phase 4: Transition to Depth (1 minute)#

"Four places this gets interesting: choosing the signal, the gap between spike rise time and provisioning time, the dependencies that don't scale, and how much headroom to buy and who pays for it. Which do you want first?"

If the interviewer has no preference, lead with the actuation gap. It's the most counter-intuitive point, and the other topics follow from it.

Phase 5: Deep Dives (25–30 minutes)#

Deep dive A: The actuation gap (6–8 min)

"Add up the delays. Metric scrape 15s, evaluation window 60s, API call and scheduling 10s, VM boot 90s, app start 30s, JIT and cache warm-up 60s, LB health checks and slow-start 30–60s. That's about 5 minutes from spike to useful capacity. A launch that goes from 100K to 250K RPS in 90 seconds is over before the autoscaler contributes anything."

Three tools for the gap, in order of preference:

1. Know it's coming and pre-scale. The event calendar and forecast floor cover launches and diurnal ramps.
2. Shorten the delay. Warm pools (pre-booted, stopped instances attach in ~30s), pre-pulled images, node headroom so pods don't wait on new nodes, and pre-warmed caches.
3. Absorb the remainder with standing headroom plus admission control that sheds the lowest-priority traffic.

"For a truly surprise spike, say a viral post, the formula is: headroom must cover (spike rate × actuation delay). If we can tolerate a 2× surprise within 5 minutes, we run at ~50%. That's expensive, so I'd do it only for tier-0 services and shed for the rest."

Deep dive B: Choosing the signal (5–6 min)

"Latency is the worst primary signal, because the most common cause of high latency is a slow dependency. Scaling out on latency then adds connections to the thing that's already struggling. I use concurrency as the primary signal and treat latency as an alert, not a scaling input."

Deep dive C: The dependencies that don't scale (6–7 min)

"Our app tier goes from 40 to 300 instances. Each has a 20-connection DB pool, so that's 6,000 connections against a Postgres primary configured for 5,000, whose sweet spot is much lower because each Postgres connection is a process. Scaling out converts an app-tier problem into a database outage."

Mitigations:

• Connection pooler (PgBouncer in transaction mode) multiplexes thousands of client connections onto ~200–500 server connections.
• Max instances derived from the dependency budget, recomputed when pool sizes change.
• Per-consumer quotas on shared dependencies, so one service's scale-out can't starve another.
• Cache and queue in front of writes to decouple app-tier elasticity from DB write capacity (see Scaling Writes).
• Third-party quotas as hard ceilings, with degraded modes when they're hit (see Degraded Mode).

Deep dive D: Headroom and who pays (5–6 min)

"Three AZs, each carrying a third. Lose one and the other two must carry 1.5× their load, so steady-state utilization can't exceed ~66% of the knee. The knee for this service is about 80% CPU-equivalent, so the target is ~0.66 × 80% ≈ 53%, call it 55%. On a $400K/month compute bill that's roughly $180K/month of 'idle.' It isn't idle. It's the price of zone failure being a non-event. Finance should see that line item with that label."

Phase 6: Wrap-Up (2–3 minutes)#

"The design has three layers: a forecast and calendar floor for what we know, reactive target tracking on concurrency for what we don't, and admission control for the gap. It's capped by a ceiling that protects dependencies that can't scale. Headroom is set by the zone-failure scenario and priced explicitly. The capacity model, meaning RPS per instance at SLO and the limiting dependency, is re-measured every quarter and after every major release."

The org close:

"Mechanically, the platform team owns the controllers. Each service team owns its capacity model and its event calendar entries. A capacity review each quarter makes idle spend and headroom visible per team. Most capacity incidents I'd expect aren't controller bugs. They're a team that didn't tell anyone about a launch, or a shared database nobody owned the budget for."

🎯 Staff Move: End with where the next outage comes from. For capacity, it's almost always a process gap (an unannounced event, a stale capacity model) or a shared dependency, rarely the autoscaler itself.

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Capacity is where engineering meets finance, and where a design meets the rest of the company. The autoscaler itself is a commodity: every cloud and Kubernetes distribution ships one. What isn't a commodity is the judgment around it. Which signal actually measures demand for this workload? Which time constants make reactive scaling useless? Which components silently won't scale? How much should the company pay to make a zone failure boring?

It's also the purest test of the Staff thesis. The L5 answer ("autoscale on CPU") is correct most of the time. The L6 answer is opinionated about the minute it isn't: the launch spike, the zone loss, the DB connection cliff. And it names who gets paged and who paid for the headroom.

1.2 The L5 vs L6 vs L7 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"How fast does demand rise compared to how fast capacity arrives, and who pays for the difference?"

• If demand rises slower than provisioning (diurnal curves over hours), reactive scaling works and the question is tuning.
• If demand rises faster and it's known (launches, sales), you pre-scale and the question is process: who files the event and who approves it.
• If demand rises faster and it's unknown, you choose between paying for standing headroom (finance pays) and shedding load (some users pay). You say which, for which traffic tier, and who signed off.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Cost efficiency. Common for internal services, batch processing, dev/test and variable workloads without strict latency SLOs. Run hot (70–90% utilization for async), scale in aggressively, use spot/preemptible instances with interruption handling, and scale to zero outside business hours. The risk is latency SLO breaches during ramps, which the team explicitly accepts.

Reliability through known peaks. Consumer products with launches, flash sales, sports events, tax deadlines and Monday mornings. The work is forecasting, calendaring, load testing and pre-scaling, and making sure every dependency on the path is sized for the event, not only the stateless tier. Failures here are usually process failures: marketing sent the email early, the forecast used last year's numbers, or the DB wasn't included in the load test. See Flash Sales.

Survival of unknown spikes and failures. Tier-0 services such as login, checkout, and core APIs other services depend on. They need standing headroom for zone and region loss, load shedding with priority tiers, and protection from retry storms. Here the design question is how the service degrades, not how it scales.

🎯 Staff Move: "These aren't mutually exclusive. They're tiers. I'd classify each service: tier-0 gets survival headroom, consumer-facing gets known-peak planning, and internal/batch gets cost efficiency. One autoscaling policy for all 200 services is how you get either an outage or a bill."

2.2 When NOT to Auto-Scale#

🎯 Staff Move: "The database is where I stop autoscaling and start capacity planning. I'll scale the stateless tier elastically and plan the stateful tier a quarter ahead, with growth tracking and a 6-month runway alert."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: Scaling Signal#

Staff default: concurrency for synchronous services, backlog-over-drain-rate for async, CPU only for CPU-bound workers. Latency is an alert, never a primary scaling input. Always pair the reactive signal with a ceiling, so that "concurrency rising because the DB slowed down" can't turn into 300 instances hammering the DB.

"Little's Law gives me the target. If one instance meets SLO at 400 RPS with 150ms average latency, it holds 400 × 0.15 = 60 requests in flight. I target 60 and scale on it. If latency doubles because the DB slows, in-flight doubles too, and the ceiling is what stops me from making it worse."

🧭 Principal Move: "I'd publish a signal standard per workload class (request/response, stream consumer, batch, GPU inference) so 200 teams don't rediscover that CPU is wrong for I/O-bound services one incident at a time."

3.2 Fault Line 2: Reactive vs Predictive#

Staff default: forecast and schedule set the floor. Reactive scaling tracks the actual signal above it. The controller takes max(floor, reactive_desired), clamped by the ceiling. A wrong forecast therefore costs money (too high) or falls back to reactive behavior (too low). It never removes the reactive path.

When to deviate: services with flat load can skip prediction. Services with extreme predictable spikes (a daily 9:00:00 batch trigger, a sports kickoff) should use scheduled capacity, since forecasting adds nothing there.

🎯 Staff Move: "Predictive scaling should only ever raise the floor. The day it's allowed to lower capacity below what reactive wants is the day a bad forecast becomes an outage."

3.3 Fault Line 3: Cold Start vs Warm Capacity#

Staff default: warm pool for VM fleets, low-priority placeholder pods for Kubernetes (~10% node headroom), pre-warming on startup (hit the top 1,000 cache keys, run a synthetic request loop until p99 stabilizes) before registering with the LB, and LB slow-start of 60s.

🎯 Staff Move: "A new instance that isn't warm is worse than no instance. It takes a full share of traffic, serves it slowly, fails health checks, and gets replaced by another cold instance. Warm-up has to finish before registration."

3.4 Fault Line 4: Headroom vs Cost#

Headroom cost ≈ fleet cost × (1 − utilization), assuming the fleet is sized to peak. Autoscaling reduces it off-peak but not at peak.

Staff default: set the target by the failure scenario you've committed to survive, then check it against the latency knee. For a 3-AZ user-facing service, ~55–60%. For async workers, 80–90% (the queue is the headroom). For tier-0 services with region-failover requirements, whatever the failover math says, even if it's 45%.

"I'd rather say '60%, because losing one of three zones multiplies load by 1.5 and our knee is at ~85%' than '60% is industry standard.' The first is a decision. The second is a habit."

🧭 Principal Move: "Headroom should be a line item with a scenario attached: '$200K/month buys AZ-loss tolerance for checkout.' Then leadership can make the trade explicitly. Most orgs pay for headroom they can't explain and skip headroom they need."

3.5 Fault Line 5: Central Capacity vs Service Ownership#

Staff default: the hybrid. Specifically:

• Platform team: autoscaling controllers, warm pools, node provisioning, load-test tooling, capacity dashboards.
• Service team: capacity model (RPS per instance at SLO, limiting dependency), scaling policy, event calendar entries, load tests before launches.
• Shared-dependency owners (DB platform, API platform): publish per-consumer budgets and enforce them.
• Capacity review (quarterly): growth vs. runway, headroom vs. cost, top risks.

🎯 Staff Move: "The failure I'd design against is the unannounced launch. The fix is process, not technology: an event calendar that marketing and product write to, and a pre-scale approval with a named owner."

4. Failure Modes & Operational Reality#

4.1 The Scale-Out Death Spiral#

t=0:       Spike: 2× traffic in 60s.
t=+45s:    p99 from 200ms to 2s. Health checks (1s timeout) begin failing on busy instances.
t=+60s:    LB ejects 30% of instances. The rest go to 140% of capacity.
t=+90s:    Autoscaler adds 60 instances. Clients retrying, so offered load is 3×.
t=+4min:   New instances register cold, get a full share, fail health checks, get replaced.
t=+5min:   New app instances add 1,200 DB connections. DB CPU 100%. All instances slow.
t=+8min:   Every instance is "unhealthy" at some point. Effective capacity near zero.

Detection: lb.healthy_host_ratio falling while autoscaler.desired_capacity rises; db.active_connections near max; client.retry_ratio > 1.5.

Mitigation: freeze scale-in and instance replacement (stop the churn); enable gateway admission control to cut offered load to known capacity; separate liveness from readiness checks (a slow instance is busy, not dead); cap scale-out at the DB budget.

Prevention: health checks that test "the process is alive," not "the process is fast"; LB slow-start; warm-up before registration; retry budgets in clients (≤ 10% extra load); circuit breakers (Circuit Breakers); ceiling from dependency budgets.

Owner: service on-call leads; platform on-call freezes the controllers; DB on-call guards the database.

4.2 Scaling on a Symptom: Latency-Driven Scale-Out into a Slow Database#

A slow query plan makes DB latency jump from 5ms to 80ms. App concurrency rises 16×, and the reactive controller (on concurrency) scales from 60 to 300 instances. The DB now has 5× the connections and is slower still.

Detection: autoscaler.scale_out_events correlated with db.query_latency_p99 rising before request rate did. The signature is capacity rising while RPS stays flat.

Mitigation: freeze scaling; fix or roll back the plan; shed.

Prevention: a flat-RPS guard: don't scale out if RPS hasn't risen more than 20% in the window, since rising concurrency with flat RPS means a dependency slowed. Also the ceiling from the dependency budget.

Owner: service team (policy), DB team (query plan).

4.3 Scale-In Flapping and Connection Drops#

A too-short stabilization window makes the fleet oscillate between 80 and 120 instances every 4 minutes. Each scale-in terminates instances holding long-lived connections (WebSockets, gRPC streams), which reconnect and spike load on the rest, which triggers scale-out.

Detection: autoscaler.direction_changes_per_hour > 4; connections.reset_total spikes aligned with scale-in.

Mitigation: lengthen the scale-in stabilization window to 10–15 min; cap the scale-in step at 10% per window; use connection draining (up to 5 min) before termination.

Owner: service team.

4.4 Capacity Ceiling Nobody Knew About#

Launch day: the fleet scales perfectly to 280 instances, and the payment provider's API rate limit of 2,000 RPS fails 30% of checkouts. Or the cloud account's vCPU quota in the region stops scale-out at 180 instances. Or the NAT gateway's port allocation is exhausted.

Detection: autoscaler.scale_out_failures{reason=quota}; third-party 429 rates; nat.port_allocation_errors.

Mitigation: emergency quota increase (hours to days, which is too slow for today); shed low-priority traffic; queue payments for async capture.

Prevention: a limits inventory per service: cloud quotas, third-party limits, NAT/IP, license counts. Checked in the pre-launch review, with alerts at 70% of every limit.

Owner: service team for the inventory; platform for cloud quotas.

4.5 Forecast Wrong by 2×#

Predictive floor built from the last 4 weeks; a holiday week cuts the actual load in half. Paying 2× for a week is a cost incident, not an outage. The opposite case, a viral event at 2× the forecast, falls back to reactive plus headroom plus shedding.

Detection: forecast.error_pct (actual/forecast − 1) tracked daily; alert when |error| > 30% for 2 hours.

Owner: platform (forecasting), service team (event calendar hygiene).

4.6 Operational Reality Matrix#

🎯 Staff Move: "When I read this matrix, the autoscaler is at fault in maybe one row. The rest are process, dependencies and client behavior. That's why I'd spend my design time on those."

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Knowing HPA YAML fields ≠ capacity judgment. Configuration fluency doesn't mean the candidate knows which signal to pick.
• Queueing theory derivations ≠ design. M/M/1 math is useful for one sentence ("latency ∝ 1/(1−ρ)"), not for ten minutes.
• "We'll use serverless" ≠ solving scaling. It moves the problem to cold starts, concurrency limits and downstream protection.
• Kubernetes cluster autoscaler internals ≠ capacity planning. Node provisioning is one delay in the chain.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• The exact target-tracking algorithm and PID tuning. One sentence is enough.
• Instance type selection details. Mention only if CPU:memory ratio matters.
• Cloud-provider-specific policy syntax.
• Detailed queueing theory beyond the knee.

6.4 Follow-Up Questions to Expect#

1. "What's your scale-in policy?" Slow (10–15 min stabilization), small steps (≤ 10%), connection draining, never below the floor.
2. "How do you avoid scaling on noise?" Evaluate over 1–3 min windows, require N consecutive breaches for scale-in, but allow a single strong breach to trigger scale-out.
3. "How do you handle multi-tenant load on shared services?" Per-tenant quotas at the gateway (Rate Limiting) so one tenant's spike doesn't drive everyone's capacity.
4. "What about stateful services?" Plan quarterly; vertical first; add read replicas ahead of need; shard deliberately (Database Sharding).
5. "How do you load test safely?" A prod-like environment with production-sized data, or production traffic shifting with automated abort thresholds.
6. "How do you forecast?" Weekly seasonality × growth trend × event multipliers; compare forecast error daily.
7. "Who decides to pre-scale for Black Friday?" The service owner files it; the capacity reviewer approves; finance sees the cost; the platform team executes.

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design auto-scaling for our API."

Drill 2: Compute the Actuation Gap#

Prompt: "Traffic doubles in 60 seconds. Walk me through the timeline of your autoscaler."

Drill 3: Pick the Signal#

Prompt: "Our service averages 25% CPU but p99 latency hits 2s at peak. The autoscaler never fires. What's wrong?"

Drill 4: The Dependency Ceiling#

Prompt: "The app tier scaled from 50 to 250 instances during a sale and the database fell over. Design so this can't happen."

Drill 5: Headroom Math#

Prompt: "Why do you run at 55% and not 80%? Finance wants 80."

Drill 6: Queue Workers#

Prompt: "Scale a fleet of workers consuming from a Kafka topic with 64 partitions."

Drill 7: Load Testing That Tells the Truth#

Prompt: "Your load test showed we could handle 3× peak. On launch day we fell over at 1.4×. Why?"

Drill 8: Build vs Buy#

Prompt: "Should we build our own autoscaler?"

Drill 9: Changing the Policy Without an Outage#

Prompt: "You want to move 150 services from CPU-based to concurrency-based scaling. How?"

Drill 10: Multi-Region#

Prompt: "We're going active-active in 3 regions. How does capacity change?"

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Incident — Launch Day Brownout#

Context: A highly promoted feature launches at 10:00. At 10:02, p99 latency is 6s and 12% of requests fail. The autoscaler shows desired capacity climbing from 80 to 240. On-call escalates to you.

Questions to Surface First:

• Is the new capacity serving, or stuck in boot/warm-up/health-check failure?
• Which dependency is at its limit: DB connections, third party, cache?
• Is offered load inflated by retries?
• Was this launch on the event calendar, and was it pre-scaled?

Typical L5 Approach: Raise max instances; increase instance size; wait for the autoscaler to catch up.

Staff Approach: Stop the churn: freeze instance replacement, since health-check failures are killing slow-but-alive instances. Enable gateway admission control to cap offered load at current real capacity, shedding low-priority traffic first. Check the DB connection count and cap max instances below the dependency ceiling. Once stable, let warm capacity join with slow-start. Then ask why the launch wasn't pre-scaled.

Principal Approach: Treats it as a planning-process failure. Institutes a launch-readiness gate: any launch with expected traffic > 1.5× baseline requires an event calendar entry, a load test at 2× forecast including dependencies, a pre-scale plan and a named approver. Adds launch readiness to the product review checklist, so marketing can't send the email without an infra sign-off.

Key Takeaway: "Known events are a process problem. If a launch needs the autoscaler to save it, the planning already failed."

What clears the Staff bar:

• Stops the death spiral before adding capacity
• Uses shedding with pre-agreed priorities
• Traces root cause to the missing pre-scale process

Deep Dive 2: Silent Failure — Headroom That Wasn't There#

Context: A zone outage lasts 40 minutes. The service was "designed for AZ loss" at a 60% target, yet it browned out for the whole outage. Dashboards showed 60% average utilization before the event.

Questions to Surface First:

• 60% of what? Averaged across the fleet, or at the hottest instances?
• Did the remaining zones have instance quota and warm capacity?
• Did traffic distribute evenly, or did one zone take the surge?
• Was the capacity unit (RPS per instance at SLO) still accurate after recent releases?

Typical L5 Approach: Raise the target headroom to 50% and move on.

Staff Approach: Finds the failures in the assumption chain. The 60% was an average, but the hottest AZ ran at 72% because of uneven LB weights. A release 6 weeks earlier made requests 25% more expensive, so the real capacity unit had dropped from 400 to 300 RPS per instance and the actual utilization vs. SLO was ~80%. And the regional vCPU quota prevented scale-out to compensate. Fixes: measure utilization against the current capacity unit, re-measured after each major release; monitor per-AZ peak, not the average; keep quota headroom ≥ 1.5× peak; run an AZ-evacuation game day quarterly.

Principal Approach: Institutionalizes headroom verification: every tier-0/tier-1 service runs a quarterly AZ-drain exercise, removing a zone's capacity in production for 30 minutes. A service that fails loses its tier-0 designation until fixed. Headroom is no longer a number in a config. It's a tested property.

Key Takeaway: "Headroom you haven't tested is a hypothesis. Efficiency regressions quietly erase it between incidents."

What clears the Staff bar:

• Questions the averaging and the capacity unit instead of just raising headroom
• Finds the quota ceiling
• Turns headroom into a tested property

Deep Dive 3: Large Customer Onboarding#

Context: A B2B platform signs a customer that will send 40% of current total traffic, ramping over 3 weeks starting next month. Their traffic is bursty: nightly batch syncs of 5M records in 20 minutes.

Questions to Surface First:

• What's the peak shape: sustained, or 20-minute bursts at a fixed time?
• Which endpoints, and what's their cost relative to the average request?
• Which shared dependencies will they hit hardest?
• Can we shape their traffic contractually (rate limits, batch windows, a bulk API)?

Typical L5 Approach: Increase max instances by 40%; scale the database vertically.

Staff Approach: Models the customer's load explicitly: 5M records ÷ 20 min ≈ 4,200 writes/s at 02:00 UTC, which is 3× the DB's current peak write rate. Rather than sizing everything for that burst, offers a bulk import API that goes through a queue drained at a controlled rate (e.g., 1,500/s over ~55 min), with per-tenant rate limits at the gateway. Pre-scales workers on a schedule for the batch window. Load-tests the bulk path at 2× their stated volume. Ramps their traffic weekly with checkpoints.

Principal Approach: Adds a commercial-capacity interface. Deals above a threshold (e.g., > 10% of traffic) trigger a capacity review before signing, and contracts include rate limits and batch windows. Capacity becomes part of the sales process, and the cost of serving a customer shows up in deal margin.

Key Takeaway: "Don't size for a customer's burst. Shape the burst, then size for the shaped load."

What clears the Staff bar:

• Quantifies the burst against current capacity
• Changes the demand curve rather than only the supply
• Stages the ramp with checkpoints

Deep Dive 4: Post-Mortem — The Retry Storm That Scaled to the Ceiling#

Context: A 90-second DB failover caused errors. After the DB recovered, the service stayed down for 25 more minutes. The autoscaler had scaled to its max of 400 instances, and offered load was 4× normal.

Questions to Surface First:

• What was the retry policy in clients and in intermediate services?
• Did retries stack across layers (client × gateway × service)?
• Why didn't load fall once the DB recovered?

Typical L5 Approach: Raise the max instances; make the DB failover faster.

Staff Approach: Identifies multiplicative retries: mobile clients retried 3×, the gateway retried 2×, and the service retried its DB calls 3×, so up to 18× amplification on failing paths. After recovery, the backlog of retries plus normal load kept the DB saturated (a metastable failure). Fixes: retry only at one layer, with retry budgets (≤ 10% extra), exponential backoff with full jitter, and circuit breakers that stop retries during failure. Add admission control keyed on DB health so the fleet sheds load quickly and lets the DB recover.

Principal Approach: Publishes an org-wide retry standard: retries at the outermost layer only, budgets enforced by the service mesh, and a required "metastability test" for tier-0 services (inject a 2-minute dependency outage and verify recovery within 2 minutes after it ends).

Key Takeaway: "Autoscaling can't outrun a retry storm. It scales the attack. Retry budgets and shedding end it."

What clears the Staff bar:

• Computes multiplicative retry amplification
• Names metastable failure
• Moves retry policy into a platform standard

Deep Dive 5: Multi-Region Expansion#

Context: The company is adding two regions to its single US-East deployment for latency and resilience. Leadership asks what it will cost and how capacity should be planned.

Questions to Surface First:

• Is the goal latency (serve users nearby) or resilience (survive region loss), or both?
• Must a region failure be invisible (full headroom), or is a degraded failover acceptable?
• Do regional peaks coincide or follow the sun?
• Where does the data live, and can each region's data tier take failover writes?

Typical L5 Approach: Replicate the current fleet 3×. Cost triples.

Staff Approach: Models regional demand curves. With follow-the-sun peaks, the global peak is ~1.4× the largest region's peak rather than 3×. For full region-loss tolerance, each region needs capacity for its own peak plus its share of the failed region's load at that hour. Computes the two postures: (a) invisible failover, where each region runs at ~55% at local peak (~1.8× current total spend), and (b) degraded failover with shedding and a 10-minute autoscale catch-up, where regions run at ~70% (~1.4× spend). Recommends (b) for most services and (a) for tier-0.

Principal Approach: Takes the decision to leadership as a business choice with prices: "Invisible regional failover costs $X/year more than degraded failover. Degraded failover means ~10 minutes of reduced functionality once or twice a year." Pairs it with a reservation strategy: commit ~60–70% of baseline for 1–3 years at a 30–50% discount, and keep the rest on-demand for elasticity.

Key Takeaway: "Multi-region capacity is set by the evacuation scenario, and follow-the-sun traffic makes it much cheaper than N× the current fleet."

What clears the Staff bar:

• Uses non-coincident peaks to cut cost
• Offers two priced postures
• Includes the data tier and quotas

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Compare demand rise time with the actuation delay, and pick schedule, predict, react, headroom or shed accordingly
• Choose a scaling signal per workload class and derive its target with Little's Law
• Compute a ceiling from dependency budgets (DB connections, third-party quotas, partitions)
• Set a utilization target from a failure scenario and the latency knee, and price it
• Describe the death spiral, scaling into a slow dependency, flapping, hidden ceilings and retry storms, each with a metric and an owner
• Design a load test that tells the truth, and explain why production traffic shifting is better
• Describe the org model: platform-owned mechanism, service-owned capacity models, event calendar, quarterly review

The Bar for This Question#

Mid-level (L4): Configures an autoscaling group with CPU targets, min/max and cooldowns. Knows horizontal vs. vertical. Treats the autoscaler as the solution.

Senior (L5): Adds scheduled scaling for known events, read replicas for the DB and a load test. The design is reasonable and handles the diurnal curve. It doesn't quantify the actuation gap, uses CPU without questioning it, and doesn't derive a ceiling.

Staff+ (L6): Frames capacity as pre-paid risk. Compares time constants, picks a demand signal, clamps the controller between a forecast floor and a dependency ceiling, covers the gap with warm pools and shedding, ties headroom to a failure scenario with a price, and puts the owner of every piece in the answer. The interviewer should learn something from the answer, whether that's the flat-RPS guard, the capacity-unit drift after a release, or the retry amplification math.

10. Staff Insiders: Controversial Opinions#

10.1 "CPU-Based Autoscaling Is Wrong for Most Services"#

The Staff position: concurrency or backlog by default. CPU for proven CPU-bound work.

Why this matters in interviews: saying "CPU at 70%" without justification is the most common L5 tell in this topic.

10.2 "Autoscaling Doesn't Handle Spikes — Headroom Does"#

The Staff position: autoscaling saves money on slow curves. Headroom, pre-scaling and shedding handle spikes.

Why this matters in interviews: reframing autoscaling as a cost tool shows you understand the time constants.

10.3 "Load Tests Lie; Production Tells the Truth"#

The Staff position: continuous production capacity measurement with automatic abort, plus pre-launch synthetic tests at 2× forecast.

Why this matters in interviews: it shows you've been burned by a passing load test.

10.4 "Scale-to-Zero Is a Dev Environment Feature"#

The Staff position: scale to zero for dev/test, internal tools and truly sporadic jobs. Keep a warm minimum for user-facing paths.

Why this matters in interviews: it shows you weigh the latency cost of a cost optimization.

10.5 "Most Capacity Outages Are Calendar Failures"#

The Staff position: an event calendar with an owner and a pre-scale approval step is worth more than any controller tuning.

Why this matters in interviews: ending on process instead of config is a clear Staff-level signal.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer makes one service scale correctly. A Principal engineer sees capacity as the company's largest controllable operating expense and its most correlated reliability risk at once. Every service's headroom, commitment level and scaling policy adds up to a portfolio: a baseline bought with 1–3 year commitments, an elastic layer bought on demand, and a headroom layer that is insurance. The L7 job is to set the rules for that portfolio (utilization targets by tier, commitment coverage, shared-dependency budgets, failure scenarios the company pays to survive) and to make those costs visible so leadership trades them deliberately.

The Org-Level Fault Line#

Central capacity authority vs. team autonomy over spend. Teams that own their budgets move fast but over-provision (fear of pages) or under-provision (pressure on cost). They also ignore shared dependencies. A central capacity group sees the whole picture but becomes a ticket queue and loses workload knowledge.

The Principal position: federated with guardrails. Teams own their capacity models and scaling policies within org-wide targets by tier. A small central capacity team (3–6 people at a 1,000-engineer company) owns forecasting, commitments, shared-dependency budgets, the quarterly review and showback. Exceptions to targets are allowed, but they're visible and priced.

🧭 Principal Move: "I don't need every team to be efficient. I need every team's inefficiency to be visible, intentional and attached to a reason. Showback does most of the work; the review does the rest."

Cost Model#

Assumptions: cloud compute ~$0.04–0.05 per vCPU-hour on-demand; commitments save ~30–50%; engineer fully loaded ~$300K/year; utilization figures are time-weighted averages.

Reading it: past growth stage, the capacity team pays for itself 10–50× over. The biggest single lever is usually commitment coverage on the baseline, not autoscaling.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Capacity & Scaling Standard v1

Scope: All production services on shared compute platforms.

MUST

1. Every service publishes a capacity model: RPS (or throughput) per instance at SLO, limiting dependency, max safe instances. Re-measured every 90 days and after any release flagged by the perf regression check.
2. Scaling policies use a demand signal appropriate to the workload class (request/response: concurrency or RPS; async: backlog ÷ drain rate). CPU only with a documented justification.
3. Max instances ≤ the ceiling derived from dependency budgets.
4. Scale-in stabilization ≥ 5 minutes; scale-in step ≤ 20% per window.
5. Any planned event > 1.5× baseline has an event calendar entry ≥ 5 business days ahead, a load test at 2× forecast including dependencies, and a named approver.
6. Tier-0/1 services pass an AZ-drain exercise each quarter.

SHOULD

• Warm pools or node headroom for services with actuation delay > 2 minutes.
• Retries only at the outermost layer, with budgets ≤ 10%.

Utilization targets (peak, vs. measured knee): Tier-0: ≤ 50%. Tier-1: ≤ 60%. Tier-2: ≤ 70%. Async/batch: ≤ 85%.

Exceptions: Requested through the capacity review; approved by the capacity lead and the service's director; expire after 2 quarters.

Success metrics: zero capacity-caused SEV-1s at planned events; org-wide peak utilization within ±10% of tier targets; commitment coverage 60–75% of baseline; forecast error < 15% at the weekly level.

What I'd Tell the VP#

"We spend about $2M a month on compute, and roughly half of it sits idle at any moment. Some of that is insurance we need, so a data center failure doesn't take us down. A lot of it is just unmanaged. I'm proposing three things: each team sees its own idle cost monthly, we set utilization targets by how critical a service is, and we commit to about two-thirds of our baseline usage for discounts. That should save $500–700K a month within two quarters. Separately, our last two launch-day outages happened because infrastructure wasn't told about the launch. A simple event calendar with a sign-off step fixes that for almost no cost."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll set utilization at 60%." Correct for one service, but no tiering, no pricing and no governance across 200 services.
• "We'll load test before big launches." This relies on heroics per event instead of continuous measurement and a readiness gate.
• "Autoscaling will reduce cost." This ignores that commitments on the baseline usually save more than autoscaling does.

Appendices

Appendix A: Scaling Mechanics in Depth#

A.1 Target Tracking#

every 15s:
    observed = avg(metric over last 60s)           # e.g., in-flight per instance
    desired  = ceil(current_ready × observed / target)
    desired  = max(desired, floor(now))            # forecast / schedule
    desired  = min(desired, ceiling)               # dependency budget
    if desired > current: scale_out(min(desired, current × 1.5))   # step cap
    if desired < current and stable_for(600s):    scale_in(max(desired, current × 0.9))

The asymmetric step caps and stabilization window are the whole safety story. Kubernetes HPA's defaults (15s loop, 300s scale-down window) encode the same idea.

A.2 Little's Law Target#

in_flight_target = rps_per_instance_at_slo × avg_latency_seconds
e.g. 400 RPS × 0.150s = 60 concurrent requests per instance

When latency rises for external reasons, in-flight rises without real demand growth. Hence the flat-RPS guard:

if concurrency_breach and organic_rps_growth_5m < 20%:
    suppress scale_out; alert "dependency slowdown suspected"

A.3 Queue Worker Sizing#

arrival_workers = arrival_rate / throughput_per_worker
drain_workers   = backlog / (throughput_per_worker × target_drain_s)
desired         = ceil(arrival_workers + drain_workers)
desired         = min(desired, partition_count, downstream_budget / per_worker_rate)

A.4 Predictive Floor#

floor(t) = seasonal_baseline(t, weeks=4)          # same hour-of-week, median
         × growth_trend                            # e.g., 1.02 week-over-week
         × event_multiplier(t)                     # from calendar
         × safety_factor                           # 1.1–1.3
         ÷ rps_per_instance_at_slo × (1 / target_util)

The floor may only raise capacity. Reactive scaling always operates above it.

Appendix B: Capacity Model Construction#

B.1 Finding the Knee#

B.2 What Goes in the Model#

B.3 Dependency Ceiling Worksheet#

Appendix C: Load Shedding & Admission Control#

Appendix D: Client Behavior#

• Retries: at the outermost layer only. Exponential backoff (base 100ms, cap 20s) with full jitter. Retry budget ≤ 10% of requests.
• Honor Retry-After on 429/503.
• Deadlines propagate. Don't do work whose caller has already given up.
• Idempotency keys on retried writes.

Appendix E: Observability#

E.1 Core Metrics#

capacity.utilization_vs_unit{service,az}     # demand ÷ (instances × capacity unit)
capacity.headroom_pct{service}               # 1 − utilization at peak
autoscaler.desired / .inservice / .pending   # gap = actuation in progress
autoscaler.scale_out_failures{reason}        # quota, capacity, image pull
autoscaler.direction_changes_per_hour        # flapping
warm_pool.available
gateway.shed_rate{priority}
forecast.error_pct{service}
client.retry_ratio                            # attempts ÷ organic requests
cost.idle_dollars{team}                       # showback

E.2 Critical Alerts#

Appendix F: Scale Evolution#

What you don't build on day one: a custom autoscaler, a forecasting ML model, a central capacity team. Do build on day one: the right signal, a ceiling, and a pre-scale habit for launches.

Appendix G: Multi-Tenancy, Fairness & Cost#

• Tenant-driven demand: per-tenant quotas keep one tenant from driving everyone's scale-out. The capacity bill follows the tenant through cost attribution.
• Shared-dependency budgets: each consuming service gets a connection/RPS budget from the dependency owner, enforced at the pooler or mesh.
• Showback: monthly per-team report of spend, peak utilization, idle dollars, and the scenario headroom buys.
• Spot/preemptible: fine for stateless batch and fault-tolerant async workers (typically 60–90% cheaper). Never for tier-0 synchronous paths without on-demand fallback.