Technologies referenced in this case study: Kubernetes · PostgreSQL · Apache Kafka · ZooKeeper & etcd · Envoy, Kong & NGINX

Related: Feature Flags · Multi-Region Active-Active · Autoscaling & Capacity · Metrics & Alerting Platform · Degraded Mode Framework · Schema Design · Build vs Buy Framework · Content Delivery Network

Reading Guide#

Organized for interview use first, reference second. This page covers getting a change from a merged commit into every production region safely, and getting it back out fast. Pod-level rollout mechanics — readiness probes, maxUnavailable, disruption budgets — live in Kubernetes. The step-by-step schema migration pattern lives in Schema Design. Separating deploy from release with runtime switches lives in Feature Flags. This page links to them rather than repeating them.

Without a staged pipeline:
t=0:       Engineer merges a change to the pricing service. CI green.
t=+8min:   Deploy script rolls all 12 regions in parallel, 25% of pods at a time.
t=+14min:  100% of pods on the new version. A rarely used currency path now throws.
t=+40min:  Error rate for EUR checkouts at 31%. Global average error rate 0.6% → 0.9%:
           under the 1% alert threshold. Nothing pages.
t=+3h:     Finance notices EUR revenue down. Incident declared.
t=+3h20m:  "What changed?" — 41 deploys in the last 4 hours across the company.
t=+3h50m:  Culprit found. Rollback runs. Old version can't read rows the new version
           wrote with a new enum value. Second outage.

With waves, evidence-based gates and pre-verified rollback:
t=0:       Change merged. Pipeline runs an N-1 compatibility test: old code reads
           new-format rows. Pass.
t=+25min:  One-box in the first wave's region (≤ 10% of that cell's traffic).
t=+40min:  Gate holds: 'POST /checkout currency=EUR' has seen 38 requests, needs 500.
t=+2h10m:  500 EUR requests seen. Canary error rate on that endpoint 30% vs 0.2% baseline.
t=+2h11m:  Automatic rollback of the one box. Change blocked from promotion.
           Owner notified with the failing endpoint and diff. Blast radius: one box,
           one cell, ~150 failed checkouts.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What the Interviewer Is Scoring#

A deployment system is not a speed question. Nobody is impressed that your pipeline deploys in eight minutes.

It is a blast-radius and time-to-undo question that tests:

• Whether you can state, as a number, how much of production a bad change can reach before it is detected
• Whether your gates wait for evidence — requests on the changed code path — or just for a clock
• Whether you know that a rollback is itself a deploy, and can fail, usually because of data written by the new version
• Whether you treat configuration, rules and content pushes as deploys, because the worst public outages were not code
• Whether the pipeline has a failure posture of its own — what happens when the thing that ships fixes is the thing that's broken

The key insight: The damage a bad change does is roughly exposure fraction × (time to detect + time to undo). Speed of deployment appears nowhere in that product. Staff candidates design every stage to shrink one of the three terms — smaller first steps, gates that detect sooner on the right signals, rollbacks that are pre-verified and fast — and treat deploy speed as the budget they spend to buy those, not the goal.

One Question, Three Levels#

Positions to Commit To#

Which Problem Are We Solving?#

Three intents produce three different systems. Name them, then commit.

🎯 Staff Move: "I'll design the service deployment pipeline — that's where the scale and the multi-region risk are. But I'll put config and content pushes on the same wave machinery with shorter bake times, because the biggest deployment outages in public post-mortems were not code deploys. Client software I'll treat as a separate problem with a shared gate engine. Tell me if you'd rather go deep on one of the others."

Where the Design Splits#

How Real Companies Built It#

Why this section belongs here: Deployment safety is unusually well documented, because the companies that got it wrong published post-mortems and the companies that got it right published their pipelines. Each entry shows a fault line from this page in production.

Amazon — Waves, One-Box Stages and Evidence-Based Bake#

Amazon's Builders' Library describes production pipelines split into waves: the first wave deploys to a single low-traffic Region one Availability Zone or cell at a time, the second to a high-traffic Region, and later waves to growing groups of Regions in parallel. Each wave starts with a one-box stage that typically serves at most 10% of the Region or AZ's requests; a typical pipeline then waits at least 1 hour after each one-box stage, at least 12 hours after the first regional wave and 2–4 hours after later waves, and the bake includes waiting for a number of data points (the article's example is at least 100 requests to a Create API). Deployments roll back automatically on alarm, avoid nights, weekends and holidays, and by default reach all Regions in about four or five business days (Amazon Builders' Library). A companion article describes the two-phase "prepare, then activate" technique for changing data formats so a rollback stays safe (Amazon Builders' Library).

Staff insight: Two numbers in that article do most of the work in an interview: "at most 10% of one AZ" as the first production exposure, and "wait for at least N requests" as the bake condition. Say both. The four-to-five-day global rollout is the price of fault line 1, paid deliberately — and the article also describes an expedited path for urgent fixes that still goes through every stage, with more review rather than fewer steps.

CrowdStrike — Channel File 291 and Unstaged Content (2024)#

CrowdStrike's root-cause analysis says that a sensor Template Type introduced in February 2024 defined 21 input fields while the integration code supplied only 20, and that the mismatch passed testing because early content used wildcard matching for the 21st field. On 19 July 2024 two new Template Instances were deployed as Rapid Response Content; one used a non-wildcard criterion for the 21st input, the Content Validator — which had a logic error — passed it, and sensors that received the new Channel File 291 performed an out-of-bounds memory read and crashed. Among the findings: each Template Instance should be deployed in a staged rollout through canary and successively wider rings, and customers should get control over when Rapid Response Content is deployed (CrowdStrike root cause analysis).

Staff insight: This is intent 2 — content, not code — on the client-software side, where rollback means touching each device. The lessons map directly onto this design: content is a deploy and needs rings; the validator must test against what the consumer actually does, not the schema's declaration; and the consumer must survive a malformed payload (bounds checks, last-known-good). In an interview, say "content that runs on a million machines gets a canary ring, no matter how urgent it is."

Cloudflare — A WAF Rule Deployed Globally (2019)#

Cloudflare's post-mortem of 2 July 2019 describes a new WAF managed rule containing a regular expression that backtracked enormously, pushing CPUs serving HTTP traffic to nearly 100% across its network. WAF rules were deployed through its distributed key-value store and reached the whole fleet within seconds, unlike its staged software releases; the rule went out at 13:42 UTC and the WAF was globally disabled at 14:07 UTC. Afterwards Cloudflare changed its procedure to stage rule rollouts the way it stages other software while keeping an emergency global path for active attacks (Cloudflare post-mortem).

Staff insight: Fault line 2 in one incident. The software pipeline was staged; the rules channel wasn't, because rules "had to be fast". The fix wasn't to make rules slow — it was to make them staged by default with an emergency lane that is explicit, audited and rare. See Content Delivery Network for the edge side.

Meta — Quasi-Continuous Release With Employee and Percentage Tiers#

Meta's engineering blog describes moving the Facebook web front end from three pushes a day — plus 500 to 700 cherry-picks a day — to a quasi-continuous system that pushes tens to hundreds of diffs every few hours, first to employees, then to 2% of production, then to 100%. By then the master branch was taking more than 1,000 diffs a day; Gatekeeper lets code releases and feature launches roll out independently, and an emergency stop button can halt a release from going further (Meta engineering).

Staff insight: At very high change volume, the unit of deployment becomes a batch of diffs, and the employee tier is a canary with unusually good error reporting. The decoupling of deploy from release (Gatekeeper) is what makes frequent deploys survivable: most behavior changes are flag flips, not binary changes. Cite it when an interviewer asks how deploy frequency and safety can both go up.

Knight Capital — One Server Out of Eight (2012)#

The SEC's order describes Knight Capital deploying new order-routing code to its SMARS servers, during which a technician did not copy the new code to one of eight servers, and no second technician reviewed the deployment; Knight had no written procedures requiring such a review. The new code repurposed a flag that, on the eighth server, triggered old "Power Peg" code still present there. Over about 45 minutes the system sent millions of orders, obtaining over 4 million executions, and Knight lost over $460 million (SEC order).

Staff insight: Two deployment lessons. First, the system must verify the fleet converged to the intended version — "deploy succeeded" must mean every target reports the expected artifact digest. Second, reusing a flag or field that old code interprets differently is a rollback-safety and mixed-fleet hazard; mixed versions exist during every rollout, so the change has to be safe for them.

Follow-Ups to Expect#

System Architecture Overview#

Reading the diagram: Three loops at three speeds. The build loop (merge → artifact, ~12 minutes) runs thousands of times a day and is a throughput problem. The promotion loop (artifact → every cell, 1–5 days depending on risk class) is a safety problem: the orchestrator moves a release one stage at a time, and the gate evaluator decides each step from metrics labeled by version. The undo loop (alarm → previous version, target ≤ 10 minutes per cell) is the one the whole design is judged by. The change log turns "what changed?" during an incident from a 30-minute search into a query. The break-glass path deliberately bypasses CI and the orchestrator's primary region.

One-Minute Recap#

Numbers to Bring#

Interview Walkthrough

The most common mistake: Candidates spend 20 minutes on the CI half — build caching, test parallelism, Docker layers, the Git branching model — and never reach the question the interviewer cares about: "A bad change is merged at 4 p.m. How much of production does it touch, how do you find out, and how fast is it gone?" Compress CI to ~5 minutes. Spend the rest on promotion, gates, rollback safety and the pipeline's own failure posture.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional scope in one breath:

"Engineers merge changes; the system builds immutable artifacts, tests them, promotes them through pre-production and then through production in stages, decides at each stage whether to continue, and rolls back automatically when a change makes things worse. It also records every production change so incident responders can answer 'what changed?' in seconds."

Then the non-functional requirements, which is where the design lives:

"Four constraints drive everything. One: blast radius — a bad change must reach no more than one cell in one region before a gate has an opinion about it. Two: time to undo — automatic rollback within 10 minutes of an alarm, and the rollback itself must be safe, which means data written by the new version can't break the old one. Three: every channel that changes production behavior — code, config, schema, rules — goes through staged rails; nothing is global-at-once by default. Four: the pipeline is tier 0 during incidents, so there's a path to ship a fix when CI is down. Scale: I'll assume 1,500 engineers, ~800 services, 12 regions with 3 cells each, ~2,500 merges a day and ~600 releases a day reaching production."

Then name the underspecified parts:

"A few things I'd confirm: monorepo or many repos? Are there client apps or agents on customer devices, or only servers? Is there a regulated environment that needs change approval records? What's the current change failure rate — is this a greenfield pipeline or a fix for a company that keeps breaking itself? I'll assume servers only, polyrepo with a shared pipeline definition, and some services that handle payments."

🎯 Staff Move: Putting a number on blast radius in the first three minutes — "one cell, one region, ≤ 10% of that cell's traffic for the first step" — tells the interviewer that you see the pipeline as a safety system. Everything you draw next can be judged against that number.

Phase 2: Core Entities & API (1–2 minutes)#

Name the nouns in 30 seconds:

• Change: change_id, repo, commit_sha, author, reviewers, type (code, config, schema, rules, flag), reversible (true, one_way), risk_class (standard, low, expedited, emergency)
• Artifact: artifact_digest (content hash), build_id, provenance (source sha, builder identity, signature), created_at — never mutated, never re-tagged
• Release: release_id, service, artifact_digest, config_version, schema_version, changes[], pipeline_id
• Stage execution: release_id, stage (preprod, n_minus_1, one_box, wave_1 … wave_5), target (region, cell), status (pending, deploying, baking, passed, held, rolled_back), started_at, gate_evidence
• Gate evaluation: stage_execution_id, alarms_checked[], endpoints_required[], requests_seen{}, canary_score, decision, decided_by (system or person)
• Change event: the immutable log entry for every production change, from any source

API (what teams and the orchestrator call):

POST /v1/releases                      { service, artifact_digest, config_version, changes[] }
  → 201 { release_id, pipeline_plan: [stages...], risk_class }

GET  /v1/releases/{id}                 → stage-by-stage status, gate evidence, current holds
POST /v1/releases/{id}/halt            { reason }          (anyone on the owning team)
POST /v1/releases/{id}/rollback        { scope: cell | region | all }
POST /v1/releases/{id}/expedite        { justification, second_reviewer }
POST /v1/freeze                        { scope, risk_classes_blocked, until, owner }
GET  /v1/changes?region=eu-west-1&since=6h   (incident responders: what changed here?)

Release creation returns the stage plan the pipeline will follow; teams see their pipeline as data, not as a script.

🎯 Staff Move: "I'm separating the change, the artifact and the release. The artifact is what we built, immutable by digest. The release is the artifact plus the config and schema versions it expects. The change carries the reversibility flag. When something goes wrong, the question 'can we roll this back?' is answered by data on the release, not by someone reading a diff at 3 a.m."

Phase 3: High-Level Architecture (≤5 minutes)#

Draw at most eight boxes:

Walk the flow in 90 seconds:

1. A merge triggers a hermetic build; unit and contract tests run; the output is an artifact identified by digest and signed with provenance.
2. The orchestrator creates a release and its stage plan from the service's pipeline definition, which inherits the org's default wave plan.
3. Pre-production: deploy to an integration environment; run integration tests against real dependencies. Then the N-1 stage: run the old binary against data written by the new one.
4. Production wave 1: one-box in one cell of a low-traffic region. The gate waits for a minimum wall-clock bake and minimum request counts on the endpoints the diff touched, comparing the one-box with baseline boxes.
5. On pass, the cell rolls 33% at a time, then the next cell, then wave 2 in a high-traffic region, then waves 3–5 widening across regions.
6. At any stage, a high-severity alarm triggers automatic rollback of that stage's scope and blocks promotion. The owner is notified with evidence.
7. Every stage transition writes a change event to the log.

🎯 Staff Move: Say out loud: "Notice the three loops: build at thousands a day is throughput; promotion over hours to days is safety; undo in under 10 minutes is the one I'll be judged on. I'm going to spend the time on the second and third." You've now spent ~9 minutes.

Phase 4: Transition to Depth (1 minute)#

"That's the happy path, and it's the Senior-level design. What makes deployment hard is the bad change that looks fine. I'd like to go deep on four things: how a gate decides it has seen enough evidence, why rollback can fail and how to make it safe, how config and content pushes ride the same rails, and what happens when the pipeline itself is down during an incident. Where would you like to start?"

If no preference: start with gates. It's where the bad change that "passed canary" lives.

Phase 5: Deep Dives (25–30 minutes)#

For each: state the tradeoff → commit → quantify → name who pays.

Deep dive 1: Gates that wait for evidence (7–8 min)

"A bake timer answers 'has enough time passed?'. The question that matters is 'has the new code done enough work for us to judge it?'. So each gate has three conditions: no high-severity alarm in the stage's scope; a minimum wall-clock floor — 60 minutes for the one-box, 12 hours for the first region — to catch slow burns like leaks; and minimum evidence — for each endpoint and message type the diff touched, at least N requests served by the new version. Then canary-versus-baseline comparison on those endpoints, with a statistical test rather than a fixed threshold."

Quantify: "Say an endpoint's baseline error rate is 0.1% and I want to catch a doubling. That's roughly 20 errors versus 40 — about 20,000 requests per side. A one-box at 10% of a cell serving 200 requests a second on that endpoint gets 20 a second: about 17 minutes. An endpoint at 2 requests a second needs 3 hours. That's why bake time is a per-endpoint calculation, and why low-traffic endpoints need synthetic traffic or a longer gate."

Who pays: "Engineers pay in latency — a change to a rare endpoint takes longer to promote. The platform team pays for the per-endpoint, per-version metrics, which roughly doubles metric cardinality for services in mid-deploy. I'd rather pay that than ship a bad change that passed on zero traffic."

Deep dive 2: Rollback safety (7–8 min)

"The artifact is immutable; the world it wrote into isn't. Rollback breaks in three ways: the new version wrote data the old one can't read — new enum values, new message fields, a new cache format; the new version ran a destructive migration; or the new version changed an external contract that clients already adopted. So: every persisted-format change ships in two deploys — first teach every reader the new format, then start writing it — and the pipeline's N-1 stage runs the previous binary against data written by the new one. Schema changes go expand → migrate → contract, as in Schema Design, and the contract step is its own release, flagged one-way."

Quantify: "Rollback time target is 10 minutes per cell: previous image pre-pulled on every node, rollout at 33% batches, readiness gates on. For a 300-pod cell that's three batches at ~2 minutes each plus detection. If rollback isn't safe, the alternative is roll-forward, which is a full pipeline run — even expedited, that's hours, not minutes."

Deep dive 3: Config and content on the same rails (5–6 min)

"Config changes are the most common cause of fast global outages because they're built to propagate in seconds. I'd keep the propagation mechanism and change the default: a config change is a release with its own stage plan — one cell, then one region, then widening — with bake measured in minutes, not hours, because the config is small and its effects show fast. Every consumer validates the payload against a schema and keeps last-known-good, so a malformed payload is rejected locally rather than crashing the process. The global push still exists for emergencies: signed, two-person, logged, and reviewed afterwards."

Who pays: "The security team pays a few minutes of propagation on a new rule. I'd agree with them on a number — say 15 minutes to 100% for standard rules, 2 minutes on the emergency lane — and that's a signed policy."

Deep dive 4: The pipeline is down during an incident (5 min)

"The worst time for the pipeline to fail is the hour you need it. Its dependencies — source control, CI runners, the artifact registry, the orchestrator — are probably in one region, and maybe affected by the same incident. So the pipeline gets a degraded mode: rollbacks don't need CI, because the previous artifact is already cached on nodes and in a regional registry mirror; a break-glass path can deploy one of the last 30 artifacts for a tier-0 service with two-person approval from an out-of-band runner; and the orchestrator's state is replicated so a region loss doesn't lose in-flight releases. We drill it every quarter by turning CI off."

Phase 6: Wrap-Up (2–3 minutes)#

"The core idea: the damage of a bad change is exposure times detection time plus undo time, so I shrank each term — the first exposure is one box in one cell, gates wait for evidence on the changed paths, and rollback is automatic and pre-verified. Config and content ride the same rails with shorter bake. The pipeline has its own degraded mode because it's what ships the fix."

The evolution closer:

"What I'd build later: automatic per-endpoint evidence requirements derived from the diff and the call graph; progressive delivery tied to Feature Flags so most behavior changes are flag ramps, not deploys; cell-based architecture so a 'cell' really is a blast-radius boundary; and change-failure-rate dashboards by team. What I'd not build: a custom CI system — I'd buy or adopt one and own the promotion layer, which is where the safety lives."

🎯 Staff Move: End on the damage formula and who owns each term. Senior candidates end with "and we'd add more tests." Staff candidates end with "the platform owns the wave plan and the gate engine, each service owns its alarms and evidence requirements, and change failure rate is a number we review monthly — because the next outage is most likely a change."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Deployment is the dependency every change has. A database design affects one system; the deployment system decides how every change to every system meets production. When it works, nobody notices — which is why its quality is invisible to Senior engineers and obvious to anyone who has run incident review for a year and watched "a deploy" appear in the root cause of most of them.

It also has a deceptive happy path. A Senior engineer can build a pipeline that ships good changes quickly on day one. The design is judged entirely by the bad changes: the one that only breaks a rare code path, the one that leaks memory for 18 hours before it falls over, the one that writes data the old version can't read, the config push that reaches every server in four seconds. None show up in a demo. Staff candidates design for them before drawing the first box.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"A change that breaks one code path used by 0.5% of requests is merged at 4 p.m. on a Thursday. Walk me through every stage it reaches, what each gate sees, when it's detected, who is paged, and what happens to the data it wrote when you roll it back."

A candidate who answers with the first exposure (one box, one cell), the evidence the gate requires on that path (and how long a low-traffic path takes to accumulate it), the comparison method (canary vs baseline, per endpoint), the rollback mechanism (automatic, pre-pulled, ≤ 10 minutes), the data question (N-1 stage, two-phase format changes) and the owner (service on-call for the alarm, platform for the gate engine) has operated a deployment system. A candidate who says "the canary would catch it" has built a pipeline that ships good changes.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Service fleet deploys. You control the machines. Changes are binaries or containers deployed to cells across regions. The design centers on staged exposure, version-labeled metrics, automatic rollback and the data-compatibility discipline that makes rollback safe. Lead time from merge to all regions is a business tradeoff, typically one to five days depending on risk class. Correctness bar: change failure rate, failed-deploy recovery time, and the absence of multi-region incidents caused by a single change. This is where most interviews go, and the rollout mechanics at the pod level are in Kubernetes.

Config, rules and content pushes. Feature configuration, rate-limit tables, WAF rules, fraud models, detection content, routing weights. These changes are designed to be fast — that's their reason for existing — and usually skip the code pipeline. They are also data that every server parses, which means a malformed payload is a fleet-wide crash with no build step in between. The design centers on schema validation against the consumer's real behavior, cell-staged propagation with short bake, consumer-side last-known-good, and an emergency lane that is explicit. Correctness bar: no payload reaches more than one cell before a health signal; every consumer survives a malformed payload.

Client and edge software. Mobile apps, desktop agents, kernel modules, firmware, browser extensions. You don't control the machines, the user decides when to update, and rollback is either slow (ship a new version and wait for adoption) or impossible (the device no longer boots). The design centers on rings (employees, beta, 1%, 10%, 100%), crash-rate gates by version, remote kill switches for every feature, and a long compatibility window because old versions live for months. Correctness bar: crash-free rate by version; time from "stop" to rollout halted; percentage of features that can be disabled remotely.

🎯 Staff Move: "These share a gate engine and a change log, and almost nothing else. Fleet deploys optimize undo time. Config pushes optimize propagation safety. Client software optimizes for the fact that undo may not exist. If you ask me to build one pipeline for all three, I'll share the evaluation and audit layers and keep the delivery mechanics separate."

2.2 When NOT to Build a Staged Deployment Platform#

And within the design, things you should not build even at scale:

• Don't build your own CI execution engine. Builds are a solved commodity; hermeticity and caching are configuration problems. Own the promotion layer.
• Don't build per-team deployment scripts. Every bespoke script is a pipeline without gates, and it will be the one used during an incident.
• Don't make the canary analysis a custom ML model on day one. A statistical comparison of canary vs baseline per metric — what Kayenta-style judges do — is enough for years.
• Don't add a human approval step to every production deploy. Approvals that happen 600 times a day become a rubber stamp; put humans on exceptions, expedites and one-way changes.

The Staff signal is knowing that the promotion and gate layer is the part worth owning, and everything under it — build execution, container runtime, rollout primitives — is worth buying or adopting. See Buy or Build: The Total-Cost Test and fault line 4.

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. Where the Design Splits#

Each fault line below follows the same shape: the options, who pays for each, the Staff default, and when to deviate.

3.1 Fault Line 1: Deploy Speed vs Bake Time#

The tension: Every hour of bake catches faults that only appear under load, after cache warm-up or after a leak accumulates. Every hour of bake also delays every fix, every feature and every security patch — and, less obviously, makes each release bigger.

The hidden cost is batch size. By Little's law, the changes in flight in a pipeline equal the arrival rate times the time a change spends in it. A service merging 6 changes a day through a 2-day pipeline has ~12 changes in flight; if releases are cut every 4 hours, each one bundles about one change. Stretch the pipeline to 5 days and cut releases daily, and each release bundles 6 changes — so when one fails, the gate tells you a batch is bad and someone has to bisect it.

The Staff default: graduated, evidence-based gates. One-box: floor 60 minutes, plus ≥ 20,000 requests across touched endpoints or synthetic coverage of each one. First region: floor 12 hours, so the change sees a full daily peak in at least one region. Later waves: floor 2–4 hours. This matches the public Amazon pattern quoted earlier. Standard changes reach all regions in about 1–2 days for most services, 3–5 days for tier 0.

"I want the first region to see a full daily peak before anyone else gets the change — that's the 12-hour floor. After that, I'm mostly buying protection against correlated failure, so later waves can move in hours. And the gate counts requests on the paths the diff touched, so a change to a rarely used endpoint doesn't pass on silence."

When to deviate:

• Security fixes and incident mitigations: expedited risk class — same stages, floors cut to ~15 minutes per wave, second reviewer required. Still not global-at-once.
• Internal tools and low-blast-radius services: shorter floors; the cost of a bad deploy is small.
• Changes behind a flag that is off: the deploy itself changes no behavior, so a shorter bake is fine; the flag ramp gets the careful staging. See Feature Flags.

3.2 Fault Line 2: Region-by-Region Waves vs Global Push#

The tension: Waves bound blast radius but leave regions running different versions for days, which complicates debugging and any cross-region protocol. A global push is consistent and fast, and makes every bad change a global outage.

The Staff default: waves, and inside each region, one cell at a time. The rule that matters: no single step may touch more than one cell in any region, so a bad change can't take a whole region down, even in the late parallel waves. The first wave goes to a low-traffic region, the second to a high-traffic one where all code paths get exercised.

Version skew is a design constraint, not a bug. During a 2-day rollout, every interaction between services, regions and replicas may involve two versions. That's why the mixed-fleet compatibility rule — new code must work alongside old code, in both directions — is part of this fault line. See Multi-Region Active-Active for cross-region replication with mixed versions.

When to deviate: a global push is justified only for changes whose absence is the outage — revoking a leaked credential, blocking an active attack — and then through the emergency path: validated, signed, two-person approved, with consumers that keep last-known-good. Cloudflare's post-2019 procedure — staged by default, global for active attacks — is exactly this split.

🎯 Staff Move: "Fast propagation is a feature for the emergency and a hazard for everything else. I'll keep the global lane, but it's a different door, with a different key, and every use is reviewed."

3.3 Fault Line 3: Automatic vs Human-Approved Rollback#

The tension: Automatic rollback reacts in minutes and occasionally rolls back a healthy change because of a noisy alarm or an unrelated dependency blip. Human-approved rollback adds judgment and 15–40 minutes of page-to-decision time at night, during which the bad change keeps serving.

The Staff default: roll back automatically, resume manually. The asymmetry is the argument: rolling back a healthy change costs minutes of delay; leaving a bad one costs the incident. The rollback decision uses two signals — the service's high-severity aggregate alarm scoped to the stage's targets, and the canary-versus-baseline verdict. A rolled-back release is held; the owner reviews the evidence and either fixes it or resumes with a justification.

Guarding against false rollbacks: compare canary against a baseline started at the same time, not against the long-running fleet, so dependency blips affect both arms equally. Require the regression to be in the canary's metrics relative to the baseline, not in the global metric. Track deploy.auto_rollback_false_positive_rate (rollbacks later resumed unchanged) and keep it under ~10%; above that, engineers start asking for overrides.

When to deviate: for changes flagged one-way (a contract migration already executed), automatic rollback is disabled for that release and the plan is roll-forward; the pipeline pages a human instead. For client software, "rollback" is a halt — stop the rollout and flip kill switches — and that also happens automatically on crash-rate gates.

3.4 Fault Line 4: One Central Pipeline vs Per-Team Pipelines#

The tension: A central pipeline platform gives every service the same safety — waves, gates, rollback, change log — and becomes a bottleneck for teams with unusual needs and a single point of failure for every change. Per-team pipelines are fast and tailored, and each one is a separate opinion about how much risk is acceptable.

The Staff default: the paved road. The platform owns the orchestrator, gate engine, change log, wave plan defaults and the break-glass path. Each team owns a pipeline definition in its own repo that inherits org defaults and adds service-specific alarms, evidence requirements and extra stages. Teams can make their pipeline stricter freely; making it looser than the org floor needs an exception with an owner and an expiry.

"Safety is a floor the platform owns. Speed above the floor is the team's choice. A team that wants to skip the first region files an exception that someone has to sign and that expires."

Monorepo note: in a monorepo, build and test are central, but deployment should still be per service. The commit is atomic; the rollout isn't. Cross-service changes still need mixed-version compatibility, because services don't reach a region at the same moment.

3.5 Fault Line 5: Freeze Policy — What Ships During an Incident or a Peak#

The tension: Change causes most outages, so freezing changes during peaks and incidents reduces risk. Freezing also blocks the fix, accumulates changes for a risky thaw, and teaches engineers to bypass the pipeline.

The Staff default: freezes restrict by risk class and scope, never by "all deploys". Rollbacks always flow. During an active incident, standard changes to the affected regions and their dependencies are held automatically, so responders aren't debugging someone else's deploy. After a freeze, the thaw is metered: the pipeline releases queued changes one batch per service per wave, not all at once.

🎯 Staff Move: "A freeze moves risk in time — it doesn't remove it. I'd rather restrict risky change classes during the peak and meter the thaw than freeze everything and ship three weeks of changes on January 3rd."

4. When It Breaks#

4.1 The Canary That Saw No Relevant Traffic#

t=0 (Thu 01:10 UTC): Release R88 of the orders service enters wave 1: one-box in
           one cell of ap-southeast-2 (lowest-traffic region, 02:00-ish local lull).
           Diff changes the bulk-cancel endpoint (0.3% of traffic globally).
t=+60min:  Bake floor reached. Global error rate on the one-box: 0.08% vs baseline
           0.09%. Bulk-cancel requests served by the one-box: 3. Gate: PASS
           (clock-based gate, no evidence requirement).
t=+13h:    Wave 1 floor reached; ap-southeast-2 fully on R88. Bulk-cancel still rare.
t=+17h:    Wave 2 reaches us-east-1 (high traffic) cell 1. Merchants run end-of-day
           bulk cancels at 17:00 local. Bulk-cancel error rate 0% → 64% in that cell.
t=+17h05m: Aggregate alarm on the service: 0.4% → 0.6%. Below rollback threshold 1%.
t=+17h40m: Merchant support tickets. Manual investigation finds R88.
t=+17h55m: Manual rollback of us-east-1 cell 1 and all of ap-southeast-2.
           ~41,000 failed bulk cancels; 1,200 merchants affected.

Why it happened: the gate measured time, not evidence. The one-box saw 3 requests on the changed endpoint, the aggregate alarm averaged a 64% failure on one endpoint into a 0.2-point blip, and nobody had told the pipeline which endpoint the change touched.

Detection: gate.evidence_requests{endpoint} versus requirement; per-endpoint canary-vs-baseline error rate; deploy.passed_with_low_evidence_total as a platform metric.

Mitigation: roll back; add the endpoint to the release's evidence requirement; replay synthetic bulk-cancel traffic against the one-box.

Prevention: evidence requirements derived from the diff — endpoints and handlers whose code changed, mapped through the service's route table — with a default floor of N requests per touched endpoint; endpoints that can't reach N in the floor time get synthetic traffic or an explicit owner waiver. Per-endpoint alarms for the top revenue paths. Schedule wave 1 so its floor overlaps the region's daily peak; the Google SRE workbook makes the same point that canaries must see enough traffic to be representative and that performance defects tend to show under heavy load (Google SRE workbook).

Owner: service team (alarms and route coverage); platform team (evidence engine and the "passed with low evidence" report).

4.2 Rollback Fails Because of a Data Change#

t=0:       Release R120 of the billing service adds payment_state = 'PARTIALLY_REFUNDED'.
           R120 writes it; R119 deserializes payment_state into a closed enum.
t=+3h:     R120 in wave 2. 18,000 rows now carry PARTIALLY_REFUNDED.
t=+3h10m:  Unrelated latency regression in R120 trips the high-severity alarm.
t=+3h11m:  Automatic rollback to R119 in us-east-1 cell 2.
t=+3h12m:  R119 pods throw on read: unknown enum value. Invoice listing 500s for
           every customer with a partial refund. Error rate 2% → 11% in the cell.
t=+3h14m:  Rollback alarm on the rollback. Orchestrator halts: rollback is now
           making things worse. Pages billing on-call and platform on-call.
t=+3h40m:  Decision: roll forward to R120 in that cell, accept the latency issue.
t=+6h:     Hotfix R121 (latency fix) through expedited lane.

Why it happened: the change was deployable forward and not backward. The artifact was immutable; the data was not. This is the classic rollback trap: the new version wrote a value the old version can't read.

Detection: the N-1 stage would have caught it — run R119 against a database snapshot after R120's integration tests wrote to it. In production: deploy.rollback_error_rate_delta (error rate after rollback vs before).

Mitigation: halt automatic rollback when the rollback's own error rate rises; roll forward with an expedited fix.

Prevention: the two-phase pattern. Deploy 1 ("prepare"): every reader accepts the new value — R119.5 maps unknown enum values to a safe default and logs them. Deploy 2 ("activate"): start writing the new value, only after deploy 1 is in every region. Amazon's rollback-safety article describes exactly this prepare-then-activate sequence (Amazon Builders' Library). For schema, expand → migrate → contract with the contract step as a separate, one-way release (Schema Design). The pipeline refuses to start an "activate" release while any target still runs a version without "prepare".

Owner: service team (format discipline); platform team (N-1 stage and the "rollback made it worse" halt).

Each box is its own release through every wave. Releases A–D can each roll back safely; E cannot, and the pipeline knows it.

4.3 The Pipeline Is Down When the Fix Is Ready#

t=0:       SEV-1: a memory leak in the session service (from yesterday's release)
           is OOM-killing pods in 4 regions every ~40 minutes.
t=+10min:  Decision: roll back to the previous release. Automatic rollback didn't
           fire: the leak passed the 12h floor in wave 1 (leak takes ~18h to OOM).
t=+12min:  Rollback request fails: the orchestrator's primary region is one of the
           4 affected, its database pods are on nodes under memory pressure.
t=+15min:  Fallback: build the old version again? CI runners are in the same
           region and their artifact cache is cold. Estimated 50 min.
t=+20min:  Someone proposes kubectl from a laptop. Incident commander says no:
           no audit, no wave, and nobody knows the exact old image digest.
t=+35min:  Break-glass runner (out-of-band, different region) deploys the cached
           previous digest to the 4 regions, one cell at a time, two-person approved.
t=+55min:  OOM kills stop. Orchestrator recovers at t=+1h20m; break-glass actions
           are imported into the change log.

Why it happened: the pipeline shared a failure domain with the services it deploys. The worst-case scenario for a deployment system — being needed most when it is weakest — is predictable and usually unplanned.

Detection: pipeline.orchestrator_availability, pipeline.rollback_request_latency_p99, synthetic "deploy a no-op release" probe every 15 minutes per region.

Mitigation: break-glass path; pre-cached artifacts on nodes and in regional registry mirrors.

Prevention: the orchestrator runs active-standby across two regions with replicated state; rollback needs no build, because previous digests are pinned in every regional mirror; the break-glass runner lives outside the main platform with its own credentials, caches the last 30 artifacts per tier-0 service, deploys only digests already in the release store, and still goes cell by cell. Quarterly game day: turn off CI and the primary orchestrator, ship a rollback and a fix. Treat a leak like this one as an argument for per-version memory-growth alerts during bake, not only error alarms.

Owner: platform team (pipeline availability, break-glass); incident commander authorizes break-glass use.

4.4 A Config Push Takes Down Every Region in Seconds#

t=0:       An engineer updates the rate-limit config: adds a rule with a regex
           intended to match bot user agents. Config service validates JSON schema.
t=+4s:     Config replicated to all 12 regions via watch streams; every API gateway
           node compiles the rule.
t=+6s:     The regex backtracks catastrophically on long user agents. Gateway CPU
           at 100% fleet-wide. Global error rate 0.1% → 78%.
t=+3min:   Pages from every region. "What changed?" — config change log shows the push.
t=+9min:   Revert pushed. Gateways recover as they pick up the revert and as
           pods restart.

Why it happened: the config channel was built for speed and treated as data, not as a deploy. Schema validation checked shape, not behavior. This is the Cloudflare 2019 shape of failure on your own infrastructure.

Detection: gateway CPU by config version; config.apply_errors_total; per-cell health after each config version.

Mitigation: revert; consumers fall back to last-known-good when a new config version fails a local health check (CPU, error rate) within 60 seconds of applying it.

Prevention: config goes through the same wave engine with short floors — one cell (5 minutes), one region (10 minutes), then widening; total ~30–45 minutes to everywhere for standard rules. Behavioral validation: compile and run the rule against a corpus of recorded requests with a CPU budget before publish; regex engines with linear-time guarantees for user-supplied patterns. Emergency global lane exists, signed and two-person.

Owner: config platform (staging, last-known-good); rule-owning team (behavioral tests).

4.5 Slow Burn Passes Every Gate#

t=0:       Release R7 of the search service introduces a cache keyed by query
           that never evicts.
t=+12h:    Wave 1 passes: memory 41% → 52%. No alarm (threshold 85%).
t=+28h:    Waves 2–3 pass. Wave 1 region at 79%.
t=+31h:    Wave 1 region pods start OOM-restarting in sequence; p99 latency 3×.
t=+33h:    Waves 1–3 regions all affected; rollback across 7 regions.

Why it happened: gates measured instantaneous health, not trends. A leak is a slope; the first region's floor was shorter than the time to failure.

Detection: per-version resource slope alarms during bake — memory growth per hour normalized by traffic, compared canary vs baseline; file descriptors, threads, connection counts.

Mitigation: roll back the oldest regions first; restart pods to buy time.

Prevention: include slope metrics in the canary comparison; keep the first region's floor ≥ one daily cycle; when a release is in late waves, the pipeline keeps watching early regions and can recall the release from every wave.

Owner: service team (resource alarms); platform (slope metrics in the default canary config).

4.6 Operational Reality Matrix#

🎯 Staff Insight: The most dangerous deployment failure is the one that passed. Every other row pages someone. A release that passed its gates with no evidence on the changed path pages nobody until customers do. That's why deploy.passed_with_low_evidence_total is a platform metric with a weekly review, not a dashboard nobody opens.

5. Scorecard#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• CI tooling fluency ≠ deployment design. Knowing every pipeline YAML keyword is table stakes; the Staff question is what happens between the first box and the last region.
• "We use canaries" ≠ canary analysis. A canary without a baseline, a statistical comparison and an evidence requirement is a delay.
• High deploy frequency ≠ safety. Frequency is good only with small batches, staged exposure and fast undo; frequency alone multiplies incidents.
• GitOps vocabulary ≠ a rollout design. A Git repo as the source of desired state is a good control-plane choice; it says nothing about waves or gates.

6. The 45 Minutes, Phase by Phase#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Build caching internals. One sentence: "hermetic builds with a remote cache; p50 12 minutes."
• Test pyramid debates. "Unit, contract and integration tests before production; production gates for the rest."
• Branching models. "Trunk-based with flags."
• Container image layering. Not the interview.
• Choosing a CI vendor. "Buy or adopt; we own the promotion layer."

6.4 Follow-Up Questions to Expect#

1. "Your canary passed. The release broke production in the second region. What did the gate miss?"
2. "The new version wrote data the old one can't read. Rollback has started. What happens next?"
3. "A config change took down every region in 10 seconds. How do you make sure it can't happen again without making config slow?"
4. "CI is down during a SEV-1. How does the fix reach production?"
5. "How long does a standard change take to reach every region, and what would you trade to make it faster?"
6. "Who decides whether a change can skip the first region?"
7. "How would you measure whether the deployment system is improving the company's reliability?"

7. Practice Rounds#

Drill 1: The Opening#

Prompt: "Design a CI/CD system for our company."

Drill 2: Core Mechanic — The Gate#

Prompt: "Exactly how does your pipeline decide a one-box is healthy enough to continue?"

Drill 3: Make It Concrete — Capacity#

Prompt: "Size the system: builds, deploy operations and the data the gates need."

Drill 4: Dependency Down — The Orchestrator Is Unreachable#

Prompt: "Your pipeline orchestrator is down for two hours. What still works?"

Drill 5: Hot Spot — One Service Deploys 60 Times a Day#

Prompt: "The web front end merges 60 changes a day and the pipeline takes a day and a half. What happens?"

Drill 6: Multi-Tenant — A Big Customer's Cell#

Prompt: "Your largest customer, 18% of revenue, lives in its own cell. Should their cell get changes first, last or in the middle?"

Drill 7: Build vs Buy#

Prompt: "Should we build our own deployment platform or use an off-the-shelf CD product?"

Drill 8: Policy Change Without an Outage#

Prompt: "You want to require a 12-hour bake in the first region for every service. Today most have 30 minutes. How do you roll this out?"

Drill 9: Cost#

Prompt: "Finance says CI costs $180K a month. Cut it."

Drill 10: Multi-Region#

Prompt: "We're going from 3 regions to 15. What changes in the deployment system?"

8. Incident Walkthroughs#

Deep Dive 1: Peak-Traffic Incident — A Deploy During the Biggest Sale of the Year#

Context: It's the evening peak of a seasonal sale. Checkout latency p99 jumps from 400 ms to 3 seconds in two regions. Twelve releases are mid-pipeline, three of them in those regions. The incident commander asks you: "Is it a deploy?"

Questions to Surface First:

• What does the change log say changed in those two regions in the last 6 hours — code, config, flags, migrations?
• Do the latency increases line up with a specific version label?
• Is anything shared between the two regions that isn't shared with healthy ones?
• Is the freeze policy in effect, and did anything bypass it?

Typical L5 Approach: Rolls back all three releases in those regions at once. Latency doesn't recover — the cause was a feature flag ramp to 50% that isn't in the deploy pipeline, and the triple rollback added churn during peak.

Staff Approach: Queries the change log first: three deploys, one config change, one flag ramp to 50% at 18:02. Latency by version label shows old and new versions equally slow — not a deploy. The flag ramp matches the onset. Flips the flag to 0%; latency recovers in 2 minutes. Freezes standard releases in those regions for the rest of the peak.

Principal Approach: Asks why flag ramps weren't subject to the peak freeze policy and why the change log was the only place they appeared together. Makes flags, config and deploys one change stream with one freeze policy, and adds 'ramp during peak' to the risk classes.

Key Takeaway: "'Is it a deploy?' is answered by version labels and one change log, not by rolling back everything that moved."

What clears the Staff bar:

• Checks the change log before acting
• Uses version-labeled metrics to rule deploys in or out
• Extends freeze to flags and config

Deep Dive 2: Silent Failure — Gates Have Been Passing on No Evidence for Months#

Context: An audit after an incident finds that 23% of releases in the last quarter passed the one-box gate with fewer than 100 requests on any changed endpoint. Three incidents trace to such releases. Nothing ever paged about it.

Questions to Surface First:

• Do gates have evidence requirements at all, or only time floors?
• Which services and which hours are most affected?
• Is the one-box stage scheduled in a region's off-peak?
• Do services' route tables let us map diffs to endpoints?

Typical L5 Approach: Raises every bake floor from 60 minutes to 4 hours. Lead time grows for everyone; low-traffic endpoints still see almost nothing in 4 hours.

Staff Approach: Adds evidence requirements derived from the diff, schedules wave 1 floors to overlap the region's peak, adds synthetic traffic generation for endpoints below N, and publishes deploy.passed_with_low_evidence_total by team.

Principal Approach: Treats "the gate is green but proves nothing" as a measurement-integrity problem across the org: every gate must report its evidence, gates without evidence are labeled as such in dashboards, and the platform's quarterly review tracks the share of releases promoted on real evidence.

Key Takeaway: "A green gate that measured nothing is worse than no gate, because it looks like proof."

What clears the Staff bar:

• Recognizes clock-based gates as the root cause
• Fixes with evidence, not longer clocks
• Makes the silent failure visible as a metric

Deep Dive 3: Large-Customer Onboarding — A Regulated Bank Requires Change Control#

Context: A bank signs a contract requiring that changes to its dedicated cell be approved, logged with a reviewer, and deployed only in a weekly window. Sales signed it. Engineering finds out at kickoff.

Questions to Surface First:

• What exactly does the contract require — approval per change or per release? Evidence of testing?
• Can a change record in our system satisfy their auditors?
• What happens to security fixes outside the window?
• How long can their cell lag the rest of the fleet?

Typical L5 Approach: Creates a separate manual process: a ticket per change, a human runs the deploy in the window. Their cell drifts weeks behind and becomes the snowflake every incident involves.

Staff Approach: Models the bank's cell as a wave with its own policy: last in its region, weekly window, approval recorded on the release by a named reviewer. The change log already has the audit trail. Security fixes use an expedited class pre-agreed in the contract. Mixed-version compatibility window extends to 2 weeks for that cell.

Principal Approach: Creates a standard "controlled cell" product tier with pricing, change window, approval semantics and security-fix terms — so sales sells a defined offering instead of bespoke promises.

Key Takeaway: "Change control is a policy on the same pipeline, not a second, manual pipeline."

What clears the Staff bar:

• Expresses the requirement as wave and policy configuration
• Bounds the cell's version lag
• Pre-negotiates security-fix terms

Deep Dive 4: Post-Mortem — Rollback Made It Worse#

Context: Last night a release was automatically rolled back after a latency alarm. The rollback raised errors from 1% to 14% in two cells for 40 minutes, because the new version had written records in a new format. You are leading the post-mortem.

Questions to Surface First:

• Did the pipeline's N-1 stage exist for this service? Did it cover this data store?
• Was the change flagged as reversible? By whom?
• Did the orchestrator detect that the rollback was worsening things?
• Why did roll-forward take 40 minutes?

Typical L5 Approach: Action item: "be careful with data format changes." Adds a checklist item to the PR template.

Staff Approach: Action items with owners: N-1 stage made mandatory for services with persisted state, covering every store the service writes; format changes must be two-phase and the pipeline blocks an "activate" release until "prepare" is in every cell; the orchestrator halts a rollback when the rollback's error rate rises; expedited roll-forward rehearsed.

Principal Approach: Adds reversibility to the change record and reports one-way changes per quarter; makes "rollback made it worse" a named incident category tracked across the org; funds a shared serialization library that tolerates unknown values by default.

Key Takeaway: "Rollback safety is tested before production, or it's a hope."

What clears the Staff bar:

• Names the data, not the binary, as the cause
• Adds a mechanical check rather than a checklist
• Teaches the pipeline to notice that rollback is making things worse

Deep Dive 5: Multi-Region Expansion — From 3 Regions to 12 With Cells#

Context: The company is expanding from 3 regions to 12 and introducing cells. Today, every deploy goes to all 3 regions in sequence with a 30-minute bake each. Leadership wants "the same speed" in 12 regions.

Questions to Surface First:

• Are cells truly isolated, or do they share databases or caches?
• How many services can tolerate a multi-day version skew?
• Do all services agree on which regions go first?
• What's the current change failure rate and lead time?

Typical L5 Approach: Keeps the serial plan with shorter bake — 12 regions × 15 minutes. Blast radius per step grows relative to bake, and slow burns now reach all 12 regions in 3 hours.

Staff Approach: Introduces an org-wide wave map: wave 1 (low-traffic region, cell by cell), wave 2 (high-traffic region), waves 3–5 parallel groups of 3, 4 and 3 regions, one cell per region at a time. Lead time ~1.5 days for standard, with an expedited class at ~3 hours. Verifies cell isolation before treating cells as blast-radius units.

Principal Approach: Tells leadership "same speed" is the wrong goal and offers the trade in numbers: lead time 1.5 days vs today's 1.5 hours, in exchange for bounding any bad change to one cell per region. Makes the wave map a company standard and aligns region launch order with it.

Key Takeaway: "More regions means a deliberate wave plan, not the old plan run faster."

What clears the Staff bar:

• Uses parallel waves with per-region cell limits
• Verifies that cells are real before relying on them
• Gives leadership a lead-time number and what it buys

9. Level Expectations Summary#

After studying this case study, you should be able to:

• State the damage of a bad change as exposure × (detection time + undo time) and show how each stage shrinks a term
• Design a wave plan by cell and region with a first exposure of one box in one cell
• Define gates with wall-clock floors and evidence requirements on the code paths a change touched
• Explain why rollback fails, and make it safe with an N-1 stage, two-phase format changes and expand → migrate → contract
• Put config, rules and content on the same rails with short bake and last-known-good consumers
• Design the pipeline's degraded mode: agent-local rollback, cached digests, a drilled break-glass path
• Set freeze policy by risk class and meter the thaw
• Decide what to buy (CI, rollout primitives) and what to own (promotion, gates, change log)

The Bar for This Question#

Mid-level (L4): Builds a working CI pipeline: build, unit tests, push an image, rolling deploy, maybe a staging environment. Doesn't consider staged exposure, rollback safety or config. Would ship good changes quickly and bad changes everywhere.

Senior (L5): Adds canaries, blue-green, health checks and automated rollback. Knows deploys cause incidents. The gap: canaries are clock-based and compared against nothing, rollback is assumed safe because artifacts are immutable, config is somebody else's problem, and the pipeline has no failure posture. The design is plausible and would pass review — and would still let a change to a rare endpoint pass canary on three requests.

Staff+ (L6): Frames deployment as blast radius and time to undo in the first five minutes. Builds waves by cell and region, gates on evidence, compares canary against baseline per endpoint, and proves rollback safety before production. Puts config on the same rails, designs the break-glass path, and sets freeze by risk class. Names who pays: engineers carry lead time, the metrics platform carries version-label cardinality, service teams own alarms and evidence. The interviewer should learn something from the answer.

10. Hot Takes#

10.1 Blue-Green Is a Rollback Story That Ignores the Database#

The Staff position: Blue-green is a fine mechanism inside a cell. It is not a rollout strategy. Pair it with staged traffic shifting and the same data-compatibility rules as any other deploy.

Why this matters in interviews: "Blue-green" is the answer that most often ends the candidate's thinking about rollback. Say what it doesn't cover.

10.2 Most Canaries Are Delays, Not Experiments#

The Staff position: A canary without a baseline and without evidence on the changed path is a timer. Name the comparison and the evidence or don't call it a canary.

Why this matters in interviews: "The canary would catch it" is the sentence interviewers attack first.

10.3 Human Approval on Every Deploy Makes Deploys Less Safe#

The Staff position: Automate the common case with gates. Spend human attention on the rare cases — expedites, one-way changes, exceptions to the floor — where it changes outcomes.

Why this matters in interviews: Proposing "a manager approves each production deploy" as a safety measure signals process thinking without systems thinking.

10.4 Config Is the Most Dangerous Code You Ship#

The Staff position: Config is code without a compiler, a test suite or a staged rollout — and it's designed to reach every server in seconds. The Cloudflare 2019 rule and CrowdStrike's Channel File 291 were both data, not binaries. Put config on staged rails with short bake, validate it against what the consumer actually does, and make every consumer survive a malformed payload.

Why this matters in interviews: Bringing up config unprompted is one of the clearest Staff signals on this question.

10.5 A Holiday Freeze Moves the Outage to January#

The Staff position: Blanket freezes stop fixes, encourage bypasses and turn the first week back into the riskiest week of the year. Freeze risky change classes, keep rollbacks and fixes flowing, and meter the thaw.

Why this matters in interviews: Interviewers ask about freezes to see whether you think about risk as a quantity that moves in time.

11. Beyond Staff: The Principal View#

Why L7 Sees This Problem Differently#

The Staff engineer designs a safe pipeline. The Principal engineer notices that the company has six ways to change production — the main pipeline, a legacy deploy script two teams still use, the config service, the feature-flag console, the database migration tool run by hand, and the security team's rule push — and that only the first has waves, gates and a change log. The L7 problem is not one good pipeline; it is change itself as a company-wide risk: one inventory of every channel that alters production behavior, one change stream that incident responders can query, one set of org metrics for how often change hurts us and how fast we recover, and a platform team that owns the paved road as a product with an SLO.

🧭 Principal Move: "I'm not going to ask how fast our pipeline is. I'm going to ask how many ways there are to change production without going through it, and what fraction of last year's incidents came through those. That tells me where the next year goes."

The Org-Level Fault Line#

One deployment platform for every change type vs a pipeline per change type.

🧭 Principal Move: "Every channel that changes production behavior must write to the change log, must respect freeze policy, and must stage through at least one cell with a health check before going wider. How it delivers — containers, watch streams, flag SDKs — is its owner's choice. New channels don't launch without that contract."

The org metrics. DORA's software delivery metrics — change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate — are the right vocabulary for the VP conversation (DORA). The Principal adds two breakdowns the averages hide: by team, and by change type (code, config, flag, migration). A company whose code change failure rate is low and whose config change failure rate is unknown has a measurement problem, not a safety record.

Cost Model#

Assumptions: fully loaded engineer ~$250K/year; cloud list prices; CI cost dominated by test execution; metrics cost includes per-version labels during deploys; estimates, not quotes.

The pricing insight: at every scale, the pipeline's dollar cost is small next to what it controls. At growth scale, one avoided multi-region outage a year typically pays for the whole platform team. The cost that leadership underestimates is engineer waiting time: 500 engineers each waiting an extra 10 minutes a day for builds is ~80 engineer-hours a day — roughly 10 full-time engineers — so build speed is a headcount line, not an infrastructure line.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

🧭 Principal Insight: The decisions I'd slow down on are the cell boundaries and the release record. Everything about gates and bake can change in a quarter; cells and the audit trail take years.

The Standard I'd Write#

RFC-REL-001: Production Change Standard
Status: Approved   Owners: Release Platform + SRE

Scope
  Every mechanism that changes production behavior: service deploys, config,
  feature flags, rules and content, schema migrations, infrastructure changes.

MUST
  1. Every production change is recorded in the change log with author,
     reviewer, artifact or payload digest, targets and timestamp.
  2. No change reaches more than one cell in any region before a health gate
     has evaluated it. Emergency global changes use the signed emergency path.
  3. Service deploys inherit the org wave map and bake floors; teams may add
     stages or lengthen floors, not remove or shorten them without an exception.
  4. Gates evaluate evidence on changed paths, not only elapsed time.
  5. Changes to persisted formats are two-phase (prepare, then activate).
     Services with persisted state pass the N-1 stage.
  6. Rollback is automatic on high-severity alarm; resuming a rolled-back
     release requires the owning team's approval.
  7. Every consumer of dynamic config keeps last-known-good and rejects
     payloads that fail validation.

SHOULD
  1. Ship behavior changes behind flags; ramp flags through the same cells.
  2. Run synthetic traffic for endpoints below evidence thresholds.
  3. Keep the last 30 artifacts of tier-0 services in break-glass caches.

Exceptions
  Filed with the Release Platform; reviewed within 5 business days;
  time-boxed to 1 quarter; VP Engineering sign-off for exceptions to MUST 2 or 5.

Success metrics
  - Change fail rate and failed-deployment recovery time, by team and change type
  - Multi-region incidents caused by a single change: 0
  - Releases promoted with low evidence: < 5%
  - Break-glass drill: quarterly, rollback shipped with CI off in ≤ 30 min

What I'd Tell the VP#

"Most of our outages start with a change, and we have six ways to change production, only one of which is staged and logged. I'm proposing one standard that every change channel must meet — staged through a single cell first, recorded in one change log, automatically rolled back on alarm — while each team keeps its delivery tools. It's about eight engineers for a year on the platform team, it adds roughly a day to how long a standard change takes to reach every region, and in exchange no single bad change can take down more than one slice of one region before we catch it. We'll report change failure rate and recovery time by team every month so you can see whether it's working. The main risk is friction from teams who own the fast channels; I'd bring them in by giving them an emergency lane rather than taking speed away."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll build a safe pipeline with waves and canaries" — correct for one channel; ignores config, flags and migrations.
• "The platform team owns deployments" — names an owner but not the contract between the platform and the teams, nor the exception process.
• "We'll track deploy frequency" — frequency without change failure rate and recovery time rewards the wrong thing.

Appendices

Appendix A: Mechanics in Depth#

A.1 Release Lifecycle#

Complete requires every target to report the expected digest — the Knight Capital lesson. A release that "finished" with one target on the old version is not complete.

A.2 Gate Evaluation#

function evaluate_gate(stage, release):
    if high_severity_alarm(stage.scope, since=stage.started_at):
        return ROLL_BACK("alarm", evidence)

    if now() - stage.deployed_at < stage.floor:
        return WAIT("floor")

    for endpoint in release.touched_endpoints:          # derived from diff + route table
        n = requests_served(endpoint, version=release.digest, scope=stage.scope)
        if n < required(endpoint):                      # default ~20K at 0.1% baseline
            if synthetic_available(endpoint): drive_synthetic(endpoint)
            elif waiver_active(endpoint, release): continue
            else: return WAIT("evidence", endpoint, n)

    verdict = canary_judge(canary=release.digest, baseline=stage.baseline_digest,
                           metrics=stage.canary_config)  # per-metric statistical test
    if verdict.critical_failed or verdict.score < pass_threshold:
        return ROLL_BACK("canary", verdict)
    if verdict.score < marginal_threshold:
        return HOLD_FOR_HUMAN(verdict)
    return PROMOTE

A.3 Choosing the Evidence Threshold#

For an error rate p and a regression you want to catch of k×p, a rough rule: you want ~20 baseline-expected errors in each arm, so N ≈ 20 / p. At p = 0.1%, N ≈ 20,000; at p = 1%, N ≈ 2,000. For latency regressions, a few thousand samples per arm usually suffice for p99 shifts of 20%+. These are planning numbers; the judge's statistical test makes the final call.

Appendix B: Data Model#

change            (change_id PK, type, repo, commit_sha, author, reviewers[],
                   reversible BOOL, risk_class, created_at)
artifact          (digest PK, build_id, source_sha, builder_id, signature, created_at)
release           (release_id PK, service, artifact_digest FK, config_version,
                   schema_version, change_ids[], pipeline_id, risk_class, created_at)
stage_execution   (id PK, release_id FK, stage, region, cell, status,
                   started_at, deployed_at, finished_at, baseline_digest)
gate_evaluation   (id PK, stage_execution_id FK, decision, reason,
                   evidence JSONB, canary_score, decided_by, at)
change_event      (event_id PK, source {deploy|config|flag|migration|infra|breakglass},
                   service, region, cell, ref, actor, at)   -- append-only
freeze            (freeze_id PK, scope, blocked_risk_classes[], start, end, owner)

The release store is small — ~600 releases a day × ~40 stage executions — and fits comfortably in PostgreSQL with a replica in a second region. The change-event stream goes through Kafka to the incident tooling and an indexed store queried by region, service and time.

Appendix C: Coordination Mechanisms#

C.1 Concurrency Rules#

• Per region: at most one cell in a non-steady state per release; at most K releases (default 3) concurrently mid-deploy in one cell, so a regression can be attributed.
• Per release: a release enters a wave only after the previous release of the same service has left it.
• Leases: each deploy agent holds a lease per cell (in etcd & ZooKeeper or the orchestrator's database), so two orchestrator instances can't deploy to the same cell.
• Agent-local rollback: the agent that started a deploy can revert it without the orchestrator if the alarm fires mid-deploy.

C.2 Quick Comparison#

Appendix D: API Contract & Client Behavior#

• Idempotent release creation: POST /v1/releases with an Idempotency-Key = artifact digest + config version; retries return the same release.
• Halt is cheap and open: any member of the owning team or the incident commander can halt; halting never needs approval.
• Expedite requires a reason and a second reviewer, recorded on the release.
• Webhooks to owners on hold, rollback and completion, with the gate evidence attached.
• Status API returns the plan, current stage, and the reason for any wait — "waiting for 14,200 more requests on POST /orders/bulk-cancel" is a message an engineer can act on.

Appendix E: Observability#

Control plane vs data plane: the pipeline is never in a request path, so its outages don't take services down — but they extend every service's recovery time. Alert on that: the no-op probe measures the pipeline's ability to deploy, which is what incident responders need.

Debugging the silent failure: every gate decision stores its evidence. When a bad release passes, the first question is "what did the gate actually see?" — and the answer is in gate_evaluation.evidence, not in someone's memory.

Appendix F: Scale Evolution#

What you don't build on day one: automatic bisection of batches, diff-derived evidence, a statistical canary judge of your own (adopt one), per-customer rings, or a cross-channel change spine. Start with waves, time floors plus a manual evidence check, automatic rollback and a change log; add the rest when the triggers in the evolution path fire.

Appendix G: Multi-Tenancy, Fairness and Cost#

• Fair CI scheduling: per-team concurrency quotas on runners with borrowing when idle, so one team's 2,000-test suite doesn't starve everyone at 10 a.m.
• Fair promotion: no team's releases block another's; per-service queues, with global limits only on cells in flight per region.
• Tenant-aware waves: large or regulated tenants in dedicated cells sit late in their region's order, with contractually defined windows (Deep Dive 3).
• Cost attribution: CI minutes and metrics overhead charged back per team monthly; the expensive teams are usually the ones with slow, unselective test suites — a conversation the platform team should start with data.
• Client-software rings: employees → beta → 1% → 10% → 50% → 100%, with crash-rate gates at each step and a remote kill switch; for agents running privileged code on customer machines, customer-controlled rings — the control CrowdStrike committed to for Rapid Response Content — are part of the product.