Technologies referenced in this case study: DynamoDB · Cassandra · PostgreSQL · ZooKeeper & etcd · Kafka

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once, then return to sections for targeted review.

With naive 3x replication, random placement, 24h repair:
t=0:       10 PB logical, 30 PB raw on 5,000 disks; storage bill 3x logical
t=+3mo:    Finance flags storage as the #2 infra line item
t=+6mo:    A rack PDU fails: 40 disks offline at once
t=+6mo:    Random placement means ~thousands of objects had 2 replicas in that rack
t=+6mo+2h: Unrelated disk failure elsewhere; 312 objects lose their last copy
t=+6mo+1d: Customer-visible data loss; "11 nines" marketing page becomes a liability

With erasure coding + failure-domain-aware placement:
t=0:       Same 10 PB on zone-aware RS(9,6), 5 fragments per AZ: ~16.7 PB raw (1.67x)
t=+6mo:    Same PDU failure: at most 1 fragment per stripe in that rack
t=+6mo:    Repair prioritizes stripes with the fewest surviving fragments
t=+6mo+2h: Second failure elsewhere: every stripe still has >= 13 of 15 fragments
t=+1yr:    Zero loss; storage bill ~53% lower than 3x

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What This Interview Actually Tests#

Blob storage is not a key-value store with big values. Everyone can map a key to a file path.

It is a durability-economics and metadata-scaling question that tests:

• Whether you separate the metadata plane from the data plane and give each the right consistency model
• Whether you can compute durability from failure rates, repair time, and placement — and know what the math leaves out
• Whether you trade storage overhead (money) against repair bandwidth and tail latency (operations) explicitly
• Whether you own the silent failures: bit rot, orphaned bytes, lost metadata, runaway lifecycle bills

The key insight: The data plane is embarrassingly parallel and almost never the interview bottleneck. The metadata plane — ordered keys, strong read-after-write, LIST on hot prefixes, versioning, and the commit protocol that ties bytes to names — is where designs break and where real systems spend their hardest engineering.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll design the general-purpose multi-tenant store, because it contains the other two as tiers. The hard parts are the metadata plane — strong consistency and ordered LIST at trillions of keys — and the durability math, which I want to compute rather than assert. Media origin and archive become storage classes with different placement and codes on the same metadata plane."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Citing specific, publicly documented object stores shows you've studied real durability engineering, not just the API.

Amazon S3 — Strong Consistency and ShardStore#

S3 is designed for 99.999999999% (11 nines) durability, stores data across a minimum of three AZs for standard classes, and moved from eventual to strong read-after-write consistency for all PUT, GET, LIST, and HEAD operations in December 2020 at no extra cost. AWS has publicly described the approach (a replicated cache-coherence "witness" that tracks metadata freshness) and published a SOSP 2021 paper on ShardStore, the Rust-based storage-node engine validated with lightweight formal methods.

Staff insight: S3 upgraded consistency after a decade of eventual consistency — because customers built workarounds (S3Guard, EMRFS consistent view) that cost more than fixing it at the source. Weak guarantees don't disappear; they move into every consumer.

Facebook — Haystack and f4#

Haystack (OSDI 2010) packed many photos into large append-only volume files so each read needed ~1 disk seek instead of several filesystem metadata seeks — metadata overhead was the bottleneck, not bytes. f4 (OSDI 2014) moved "warm" blobs to Reed-Solomon (10,4) within a region plus XOR across regions, cutting effective replication from 3.6× to ~2.1× for that tier.

Staff insight: Hot/warm separation is a durability-and-cost decision driven by access-age curves. Most bytes stop being read within weeks — the tiering boundary follows the data.

Microsoft Azure Storage — Local Reconstruction Codes#

Azure's LRC (USENIX ATC 2012) adds local parities over subsets of data fragments so the common case — one fragment lost — repairs by reading 6 fragments instead of 12, while keeping ~1.33× overhead and durability comparable to 3× replication.

Staff insight: The code was designed around the repair path, not the storage overhead. Repair bandwidth is the hidden cost of erasure coding, and it's paid continuously.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: A PUT writes data fragments first (step 2) and commits the metadata version last (step 3); the commit is the single point at which the object becomes visible. The metadata plane is small — ~hundreds of bytes per object — but strongly consistent and range-partitioned for ordered LIST. The data plane is enormous, append-only, and tolerant of losing any disk at any time. The background plane — repair, scrub, GC, lifecycle — is where durability is actually earned, day after day.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 15 minutes on REST endpoints and a capacity table, draw "metadata DB + storage nodes," say "3 replicas," and never reach the commit protocol, the durability math, or LIST. Compress the API to 2 minutes. The level is decided in the planes split and the durability argument.

Phase 1: Requirements & Framing (2–3 minutes)#

Functional scope in one sentence:

"A multi-tenant object store: PUT, GET (with byte ranges), DELETE, and ordered LIST over a flat key namespace inside buckets, with versioning and lifecycle policies."

Then the non-functional requirements that shape the design:

"The two numbers I'll design around are 10¹² objects — that makes metadata a distributed database problem — and 11 nines, which I want to derive rather than assume."

Phase 2: Core Entities & API (1–2 minutes)#

Bucket        { bucket_id, tenant_id, region, versioning, lifecycle_rules, policy, storage_class_default }
ObjectVersion { bucket_id, key, version_id, size, etag, storage_class, created_at,
                is_delete_marker, manifest_ref, checksum, encryption_key_ref }
Manifest      { version_id, chunks: [ {chunk_id, offset, length, stripe_id} ] }
Stripe        { stripe_id, code: RS(9,6), fragments: [ {node_id, extent_id, offset, crc} x 15 ] }
Upload        { upload_id, bucket_id, key, parts: {n -> {etag, size, chunk_refs}}, initiated_at }

Public API (HTTP):

PUT    /{bucket}/{key}                       -> 200, ETag, x-version-id
GET    /{bucket}/{key}  Range: bytes=a-b      -> 206
DELETE /{bucket}/{key}                       -> 204 (delete marker if versioned)
GET    /{bucket}?list-type=2&prefix=&continuation-token=&max-keys=1000
POST   /{bucket}/{key}?uploads               -> upload_id
PUT    /{bucket}/{key}?partNumber=n&uploadId= -> part ETag
POST   /{bucket}/{key}?uploadId=              -> complete (atomic)

"The key design decision: the ObjectVersion row is the only thing that makes bytes visible. Stripes and chunks can exist without it — that's garbage, collected later. The reverse must never happen."

Phase 3: High-Level Architecture (≤5 minutes)#

Staff candidates spend under 5 minutes here. Draw the two planes, the write order, and move on.

Say it in four sentences:

1. "Stateless API servers authenticate, rate-limit per tenant, and stream the body into chunks — say 8 MB each — erasure-coded into stripes."
2. "Placement puts at most one fragment of a stripe in any rack and spreads fragments across three AZs. One catch: RS(10,4) over three AZs puts 5 fragments in some AZ, which exceeds 4 parity — so for AZ-loss survival I'd pick a code whose parity covers the largest AZ share, like RS(9,6) at 1.67×, or a zone-aware LRC."
3. "After fragments are durable, the API server does a conditional write to the metadata partition that owns (bucket, key). That commit is the linearization point — GET, HEAD, and LIST see it immediately."
4. "Repair, scrubbing, GC, and lifecycle run in a background plane with their own budgets."

🎯 Staff Move: Sentence 2 is where candidates get caught. RS(10,4) over 3 AZs puts ≥ 5 fragments in some AZ, so losing that AZ loses the stripe. The rule: parity count ≥ the largest per-AZ fragment count. RS(9,6) at 5-5-5 costs ~1.67× instead of 1.4× — that 0.27× is the price of surviving an AZ. Say that tradeoff out loud; interviewers write it down.

Phase 4: Transition to Depth (1 minute)#

"The API and the byte path are straightforward. Where I'd like to spend time: first, durability — the math, the code choice, and the correlated failures the math leaves out; second, the metadata plane — strong consistency, ordered LIST, and hot prefixes at 10¹² keys; third, the lifecycle of bytes — multipart uploads, deletes, GC, and tiering, which is where money and data actually get lost. Which would you like first?"

Default order if they don't choose: durability → metadata → lifecycle.

Phase 5: Deep Dives (25–30 minutes)#

Deep dive 1 — Durability math (8–10 min).

• Per-disk annual failure rate λ ≈ 1.5%/yr; repair window T. Loss needs m+1 overlapping failures in a stripe of n fragments.
• Approximate annual loss probability per stripe: P ≈ n·λ · C(n−1, m) · (λT)ᵐ.
• 3× replication (n = 3, m = 2), T = 6 h: 3 × 0.015 × 1 × (0.015 × 6.85×10⁻⁴)² ≈ 4.8×10⁻¹² — ~11 nines.
• Same with T = 24 h: ≈ 7.6×10⁻¹¹ — ~10 nines. Repair time enters squared.
• RS(10,4), T = 6 h (for scale; RS(9,6) is lower still): 14 × 0.015 × C(13,4) × (1.03×10⁻⁵)⁴ ≈ 1.7×10⁻¹⁸. The independent-failure math stops being the constraint.
• "So once I'm on a wide code with fast repair, the durability budget is spent on correlated failures — AZ events, firmware bugs, bad deploys, and operator deletes. That's what I design for next."

Deep dive 2 — Metadata plane (8–10 min).

• Range-partitioned by (bucket_id, key), each partition a Raft/Paxos group of 3–5 replicas across AZs, ~1–10 GB per partition, auto-split at size or load thresholds.
• PUT = conditional insert of a new version row; GET reads the latest non-delete-marker version from the partition leader (or a follower with a read lease).
• LIST = ordered range scan with continuation token = last key returned; pages of 1,000.
• Hot prefix: sequential keys (logs/2026-09-30T12:00:00/...) concentrate writes on the last partition; split-on-load moves the boundary, but monotonically increasing keys always hit the tail. Mitigation: split heat detection + guidance to prefix with a hash; the platform can't fully fix a key design.

Deep dive 3 — Lifecycle of bytes (6–8 min).

• Multipart: parts are independent chunks with their own stripes; complete commits one version row whose manifest references part chunks — no data copy.
• Abandoned uploads: bytes exist without a version row. Lifecycle rule "abort incomplete uploads after 7 days" plus a platform-wide backstop.
• Delete: writes a delete marker (versioned) or tombstone; GC reclaims chunks after a delay (e.g., 24–72 h) so a mistaken delete or metadata bug is recoverable.
• Tiering: lifecycle moves versions to cheaper classes by age; the metadata row changes its storage_class and manifest atomically after bytes are rewritten.

Phase 6: Wrap-Up (2–3 minutes)#

"To summarize: two planes; data written first and metadata committed last so a crash leaves only garbage, never a broken name; erasure coding across failure domains with repair time as the main durability lever; range-partitioned metadata for strong consistency and ordered LIST; soft deletes and delayed GC because humans and bugs are the biggest loss risk. Next I'd build cross-region replication as a per-bucket policy, object lock for compliance, and cost visibility per tenant so lifecycle rules actually get written."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Almost every system design ends with "and we store the files in S3." This prompt turns that black box inside out. It tests whether a candidate understands the difference between a system that is correct (every PUT readable) and one that is durable over a decade (every PUT readable after 10 years of disk failures, deploys, migrations, and operator mistakes). The second property is not a feature of any component; it emerges from placement, repair, scrubbing, and process.

It's also an economics question. Storage is a line item that grows monotonically. The candidate who says "3×" at 100 PB has just committed ~133 PB of unnecessary disks versus ~1.67× zone-aware EC — tens of millions of dollars a year — and the Staff answer is expected to notice.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"What is the most likely way we lose a customer's object — and what stops it?"

Answering honestly moves the conversation away from disks. At scale, with a wide code and fast repair, the independent-disk path is ~10⁻¹⁸. The likely paths are: a deploy that corrupts metadata; a GC bug that deletes live chunks; a placement bug that puts two fragments in one rack; a firmware issue that fails 3% of one drive model in a week; a customer's own script deleting a prefix. Each needs a named defense — canarying metadata changes, GC with a delay and a reference-count audit, placement verification in the scrubber, drive-model diversity, versioning and object lock.

🎯 Staff Move: "Once the math says the disks aren't the risk, I spend the durability budget on software and humans: delayed GC, versioning, placement audits, and staged deploys of anything that can touch metadata."

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Intent 1 — General-purpose multi-tenant store. Unknown workloads from thousands of tenants: data lakes doing LIST over millions of keys, ML jobs reading at 100 GB/s, backup tools writing TB-sized objects, web apps serving 10 KB avatars. The design must be strongly consistent, isolate noisy tenants, and offer tiers because one price point can't serve all of them. Correctness bar: read-after-write, designed durability, per-tenant fairness.

Intent 2 — Media/CDN origin. Write-once, read-many, heavy-tailed popularity. Most reads are served by a CDN; the origin must survive miss storms (a viral video, a CDN purge) and keep $/GB low for a long tail nobody reads. Small-file packing (Haystack-style) cuts metadata cost. Correctness bar: durability plus availability of popular objects.

Intent 3 — Backup/archive. Writes are large and sequential; reads are rare and often all-at-once during disaster recovery. Wide codes, dense disks or tape, and retrieval measured in hours are fine — until a restore happens under pressure. Correctness bar: decades-long durability and a restore-time objective someone actually tested.

2.2 When NOT to Build Blob Storage#

🎯 Staff Move: "If the question were 'should we build this' at a normal company, the answer is no — buy it and spend the effort on lifecycle policy and cost governance. I'll design it as if we're the provider."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: Replication vs Erasure Coding#

The tension: Replication buys simplicity, fast reads, and cheap repair with storage money. Erasure coding buys storage money with repair bandwidth, CPU, and tail latency.

Staff default: Size-based hybrid. Objects < ~1 MB are packed into larger replicated or EC'd containers (Haystack-style) so tiny objects don't each become a stripe; objects ≥ 1 MB are chunked (e.g., 8–64 MB) and erasure-coded with a zone-aware code. Repair traffic is budgeted as a fraction of cluster bandwidth (e.g., ≤ 10–20%) with priority by fragments remaining.

The hidden cost to say out loud: Losing a 24 TB disk under RS(9,6) means reconstructing 24 TB by reading ~9 × 24 = 216 TB across the cluster. Declustered placement spreads that over thousands of disks, but it's continuous background load — at ~1.5% AFR and 10,000 disks, that's ~150 disk failures a year, ~3 a week.

When to deviate: A latency-critical small-object store (avatars behind no CDN) can stay replicated. Archive classes go wider (e.g., RS(17,3) inside an AZ plus cross-region copies) because repair speed matters less than $/GB.

3.2 Fault Line 2: Strong vs Eventual Metadata Consistency#

The tension: Strong consistency needs a single authority per key (consensus, leader leases); eventual consistency lets any replica answer and pushes anomalies onto every consumer.

Staff default: Strong read-after-write for single-key operations and LIST, via per-partition consensus with the metadata commit as the linearization point. No multi-key transactions; offer conditional writes (If-Match, If-None-Match) for single-key compare-and-swap. Caches in the front end are validated against a per-partition version (a witness-style freshness check) rather than TTLs.

🎯 Staff Move: "Weak consistency doesn't remove the cost — it moves it into every consumer. With thousands of tenants, I'd rather pay it once in the metadata plane."

3.3 Fault Line 3: Durability vs Write Latency#

The tension: When do we say 200 OK? Every additional durable copy before the ack adds latency; every copy after it is a loss window.

Staff default: Ack after a write quorum that spans at least 2 AZs and still survives an AZ loss plus one disk (e.g., ≥ 13 of 15 fragments in RS(9,6) with ≥ 4 per AZ). Missing fragments are enqueued for immediate repair. Never ack before data is fsynced somewhere durable — "written to page cache" is not durable.

When to deviate: A reduced-redundancy or single-AZ class can ack faster; price and name it so customers choose knowingly.

3.4 Fault Line 4: Flat vs Hierarchical Namespace#

The tension: Flat keys scale linearly and partition cleanly; directories give atomic rename and cheap listing of children, but require a tree with cross-partition operations.

Staff default: Flat namespace for the general-purpose store. Document that rename is not atomic and push analytics engines toward commit protocols that write a manifest (table formats like Iceberg/Delta commit by writing a metadata file, not renaming). Offer hierarchical as a separate bucket type only if analytics demand justifies a second engine.

Hot prefixes are the practical fault line inside flat namespaces: sequential keys hit one range partition. Split-on-load helps for spread-out heat; monotonically increasing keys always write the tail partition. Publish key-design guidance and expose per-prefix throttling (503 SlowDown) instead of letting one tenant's hot tail slow its neighbors.

3.5 Fault Line 5: Automatic vs Customer-Driven Tiering#

The tension: Cheaper tiers save 50–95% on $/GB-month but add retrieval cost, retrieval latency, and minimum-duration charges. Who decides when data moves?

Staff default: Automatic tiering only between tiers with the same access latency (monitor per-object access, move after 30–90 days idle); archive tiers with hours-long retrieval are strictly opt-in via lifecycle rules, because the victim of a surprise 12-hour restore is a customer mid-incident. Every bucket shows cost by tier and "bytes not read in 90 days" in its dashboard.

4. Failure Modes & Operational Reality#

4.1 Correlated Failure — The Bad Drive Batch#

t=0:       Drive model X (18% of fleet) hits a firmware bug at ~30K power-on hours
t=+2d:     Disk failure rate for model X: 1.5% AFR -> ~40% annualized
t=+2d:     storage.disk_failures_per_hour 3x baseline; repair queue growing
t=+3d:     storage.stripes_at_min_redundancy rises from 0 to 1,200
t=+3d:     Page: stripes with <= 1 spare fragment > 100
t=+3d+1h:  Repair re-prioritized: fewest-surviving-fragments first; bandwidth cap 10% -> 35%
t=+4d:     Placement excludes model X for new writes; proactive drain begins
t=+3wk:    Model X drained; zero loss

• Detection: storage.disk_afr{model}, storage.stripes_by_surviving_fragments, repair.queue_age_p99.
• Blast radius: Any stripe with multiple fragments on model X. Placement diversity by drive model (not just rack/AZ) caps it.
• Mitigation: Priority repair, raise repair bandwidth, stop placing on the bad model, proactive drain.
• Prevention: Treat drive model and firmware version as failure domains in placement; stage firmware updates by cohort.
• Owner: Storage SRE; hardware team for vendor escalation.

4.2 The Metadata Hot Partition#

t=0:       Tenant launches IoT ingest writing keys iot/{timestamp}/{device}
t=+10min:  All writes land on the last range partition of the bucket
t=+12min:  metadata.partition_qps{p=bucket-7:tail} = 18K vs split threshold 5K
t=+13min:  Auto-split: new boundary, but monotonic keys keep hitting the new tail
t=+15min:  Partition leader CPU 95%; PUT p99 for that tenant 40ms -> 2s
t=+15min:  Other tenants unaffected (partitions are per-bucket ranges)
t=+20min:  Tenant receives 503 SlowDown; SDK backs off

• Detection: metadata.partition_qps, metadata.split_rate, api.throttled_total{tenant}.
• Blast radius: One tenant's hot prefix — if partitions are per bucket and leaders are spread. If many hot tails share a leader host, collateral damage.
• Mitigation: Throttle with 503 + Retry-After; tenant adds a hash prefix; temporary pre-split of the key range.
• Prevention: Key-design guidance, per-prefix throttling, leader placement spreading.
• Owner: Metadata team for platform; tenant for key design.

4.3 The GC Bug — Deleting Live Data#

The most dangerous failure in any blob store: a garbage collector that decides live chunks are orphans.

t=0:       Deploy changes manifest serialization for multipart objects
t=+1h:     GC's reference scan can't parse new manifests; treats their chunks as unreferenced
t=+1h:     GC marks 2.1M chunks for deletion; grace period 72h starts
t=+6h:     Audit job: gc.marked_bytes_ratio 0.03% -> 1.4% of cluster (anomaly)
t=+6h:     Page; GC deletion halted by kill switch
t=+8h:     Root cause found; marks cleared; zero bytes actually deleted

• Detection: gc.marked_bytes_ratio vs baseline, independent reference-count audit (a second implementation that samples chunks and verifies references).
• Blast radius without delay: Permanent loss of every multipart object written after the deploy.
• Mitigation: GC kill switch; grace period before physical delete; rate limit on deletions per hour.
• Prevention: GC deletes only after two independent scans agree; manifest format changes are backward-compatible and canaried; physical deletion capped at a fraction of cluster per day.
• Owner: Storage team; GC changes require a second reviewer from the durability owner.

🎯 Staff Move: "The garbage collector is the most dangerous component we own, because it's the only one designed to delete data. I want a delay, a rate limit, a kill switch, and an independent audit on it."

4.4 Silent Corruption — Bit Rot Found Late#

• Symptom: A customer GET fails checksum on an object last read 14 months ago; reconstruction succeeds from parity.
• Root cause: Latent sector errors; one fragment corrupted, never read.
• Detection: scrub.checksum_failures, scrub.cycle_age_max (time since each disk was fully read), get.reconstructed_reads_ratio.
• Mitigation: Scrubber rewrites the fragment on detection.
• Prevention: Scrub cycle ≤ 2–4 weeks; end-to-end checksums computed at the client/API and verified at every hop — not just disk-level CRCs.
• Owner: Storage SRE owns the scrub SLO.

4.5 Multipart Orphans — The Invisible Bill#

• Symptom: A tenant's bill includes 800 TB not visible in any LIST.
• Root cause: A backup tool starts multipart uploads, crashes, restarts with a new upload ID; old parts accumulate for 2 years.
• Detection: multipart.incomplete_bytes{tenant}, age histogram of open uploads.
• Mitigation: Abort uploads older than N days; notify the tenant before abort.
• Prevention: Platform default lifecycle rule "abort incomplete uploads after 7 days" on new buckets; incomplete bytes shown in billing.
• Owner: Tenant for the tool; platform for defaults and visibility.

4.6 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Reed-Solomon math fluency ≠ storage design. Knowing Galois fields doesn't show you know where to put the fragments.
• Big capacity numbers ≠ scale thinking. "140 PB raw" is arithmetic; "a 24 TB disk rebuild reads ~216 TB cluster-wide" is scale thinking.
• Naming Ceph/MinIO ≠ design. Tools implement choices; the interview is the choices.
• "We use checksums" ≠ integrity. Where they're computed (end-to-end vs per-disk) is the point.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Reed-Solomon encoding math beyond "any k of n."
• HTTP details, signing algorithms — "standard signed requests."
• Disk filesystem internals — "append-only extents on raw devices."
• CDN design — a separate problem; see CDN & Edge Caching.

6.4 Follow-Up Questions to Expect#

1. "Walk me through a GET when 2 of the needed fragments are unavailable."
2. "How does LIST stay consistent while a partition is splitting?"
3. "What happens if the client retries a PUT whose first attempt actually succeeded?"
4. "How would you implement object lock (WORM) for compliance?"
5. "How long does it take to re-protect after an AZ comes back, and do you repair during the outage?"
6. "How do you migrate 50 PB to a new storage-node format without downtime?"
7. "How would you bill tenants fairly for requests vs bytes?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design S3."

Drill 2: Compute the Durability#

Prompt: "Prove 11 nines."

Drill 3: The PUT Commit Protocol#

Prompt: "The bytes are written but the metadata write times out. What does a GET return?"

Drill 4: LIST at Trillions of Keys#

Prompt: "How does LIST prefix=logs/2026/ work with 10¹² objects?"

Drill 5: The Hot Prefix#

Prompt: "One tenant writes 50K PUT/s to keys starting with the current timestamp. What happens?"

Drill 6: Multipart Upload#

Prompt: "A customer uploads a 2 TB file over a flaky connection."

Drill 7: Small Objects#

Prompt: "80% of objects are under 16 KB. Does erasure coding still work?"

Drill 8: Build vs Buy#

Prompt: "We run 8 PB on-prem. Build our own object store?"

Drill 9: Changing the Storage Format Without Losing Data#

Prompt: "We need to migrate 50 PB from 3× replication to erasure coding."

Drill 10: Cross-Region Replication#

Prompt: "A tenant needs a copy in another region for disaster recovery."

8. Deep Dive Scenarios#

Deep Dive 1: Peak Traffic — The Viral Object Miss Storm#

Context: A video embedded in a news story goes viral. The CDN in front of the media bucket is purged by an unrelated config change. 400K req/s hit the origin for one 200 MB object and ~5K related objects. Origin p99 GET climbs from 80 ms to 9 s, affecting every tenant in the cell.

Questions to Surface First:

• Is this one object or broad traffic? Which component saturates — API servers, the metadata partition, or the storage nodes holding those fragments?
• Why did the CDN purge, and can we restore the cache?
• Are other tenants in the same cell affected, and why?

Typical L5 Approach: Scales out API servers. The bottleneck is the 15 storage nodes holding that object's fragments — each serving hundreds of Gbps — so more API servers add more load on the same disks.

Staff Approach: Identifies a hot-object problem, not a capacity problem. Adds a hot-object cache tier in the front end (in-memory, keyed by version ID so it's always consistent), request coalescing so concurrent GETs of one object share one backend read, and per-tenant request isolation so the media tenant's storm doesn't consume shared API capacity. Works with the CDN team to restore caching with a staggered warm-up.

Principal Approach: Asks why a CDN purge could be global and instant. Pushes for purge rate limits and staged purges as a CDN platform standard, and for cell-based isolation in the storage front end so one tenant's storm has a bounded blast radius by construction.

Key Takeaway: "Immutable versions make caching trivially consistent — use that. The bottleneck of a hot object is its fragments, not the API tier."

What clears the Staff bar:

• Locates the real bottleneck (fragment holders)
• Uses version IDs for safe caching
• Contains blast radius per tenant

Deep Dive 2: The Silent Failure — Scrubber Stopped Six Weeks Ago#

Context: During a routine review you notice scrub.cycle_age_max is 44 days against a 14-day SLO. Nobody was paged. A config change moved the scrubber to a lower-priority scheduling class and it's been starved by repair and transcode jobs.

Questions to Surface First:

• How many fragments haven't been verified within SLO, and on which disks?
• Were there latent errors in the last completed scrubs, and at what rate?
• Why did a 3× SLO violation not page?

Typical L5 Approach: Restores the scrubber's priority and lets it catch up.

Staff Approach: Restores priority, then computes exposure: with a historical latent-error rate, how many stripes might now have hidden bad fragments plus a missing one? Prioritizes scrubbing stripes that already have a missing fragment (where a hidden error would mean loss) and disks with elevated SMART errors. Adds a page on scrub.cycle_age_max > 1.5× SLO and a floor on scrubber bandwidth that no scheduler class can take away.

Principal Approach: Recognizes a class of problem: background durability jobs (scrub, repair, GC audits) have no customer-visible symptom when they stop. Creates a "durability SLO" review for all background jobs, with dead-man alerts that fire when a job stops reporting, and reports these SLOs to leadership monthly like availability.

Key Takeaway: "Durability jobs fail silently. Alert on their absence, not just their errors."

What clears the Staff bar:

• Computes exposure before declaring victory
• Prioritizes by risk, not by order
• Fixes the alerting class, not just the instance

Deep Dive 3: Large-Customer Onboarding — 5 PB and 20 Billion Objects#

Context: An analytics customer will migrate 5 PB in 20B objects over 60 days, then run jobs that LIST ~200M keys per hour and read at 200 GB/s bursts.

Questions to Surface First:

• What's their key layout? Sequential, hashed, partitioned by date?
• Do they need LIST or could an inventory manifest replace it?
• Peak write rate during migration? Will they use multipart for large files?

Typical L5 Approach: Checks total capacity and adds storage nodes.

Staff Approach: Capacity is the easy part. Reviews key design with them (date-partitioned with a hash component), pre-splits their metadata ranges based on expected key distribution, provides daily inventory exports so their jobs stop LISTing 200M keys an hour, sets per-tenant request limits appropriate to their tier, and places their data across enough storage nodes that 200 GB/s bursts don't concentrate. Migration throttled to protect other tenants.

Principal Approach: Turns this into a repeatable "large tenant onboarding" program with a checklist (key design review, pre-split, inventory, limits, cost forecast) and a capacity reservation contract. Negotiates pricing that reflects metadata and request costs, not just bytes.

Key Takeaway: "Large tenants break metadata, not disks. Onboard their key design, not just their bytes."

What clears the Staff bar:

• Focuses on metadata and access patterns over raw capacity
• Replaces expensive LIST with inventory
• Protects other tenants during migration

Deep Dive 4: Post-Mortem — A Tenant Lost 3% of a Bucket#

Context: A tenant reports 3% of objects in a bucket return 404. Investigation shows their own IAM role, compromised via a leaked CI credential, issued DELETE on 1.2M keys. Versioning was off. GC ran after 24 hours. The data is gone.

Questions to Surface First:

• Was the platform working as designed? (Yes.)
• What defaults allowed an unversioned production bucket?
• Why could one credential delete 1.2M objects without any friction?

Typical L5 Approach: "Customer error; recommend they enable versioning."

Staff Approach: Accepts that "working as designed" isn't a defense when the design makes a common mistake permanent. Changes defaults: versioning on for new buckets; anomaly detection on delete rate per bucket (1.2M deletes in 10 minutes vs a baseline of 100/day) that alerts the tenant; a longer GC grace period for bulk-deleted data (e.g., 7 days) during which support can restore.

Principal Approach: Frames it as a platform-level safety posture: soft-delete by default, object lock for critical data, deletion requiring separate credentials for production buckets, and a recommended cross-account backup. Writes the "data protection defaults" standard and measures adoption across all buckets.

t=0:       Leaked credential used from unfamiliar IP
t=+3min:   DELETE rate on bucket: 2,000/s (baseline ~0.001/s)
t=+10min:  1.2M objects deleted; no alert exists for delete rate
t=+24h:    GC reclaims chunks
t=+3d:     Tenant notices missing files from user reports

Key Takeaway: "The biggest durability risk is a valid DELETE. Design defaults so the common mistake is recoverable."

What clears the Staff bar:

• Refuses 'working as designed' as a closing statement
• Changes defaults, not just documentation
• Separates security and storage responsibilities cleanly

Deep Dive 5: Multi-Region Expansion — Launching a New Region#

Context: The company is launching the object store in a new region with 3 AZs. Initial demand is 2 PB; tenants want a global namespace for bucket names and cross-region replication from day one.

Questions to Surface First:

• What must be global (bucket-name uniqueness, IAM) vs regional (metadata, data)?
• Is the new region's metadata plane independent of existing regions?
• Can we launch with a smaller cluster without weakening durability?

Typical L5 Approach: Extends the existing metadata cluster to the new region for a single global namespace.

Staff Approach: Keeps every region's metadata and data planes fully independent — a region failure must not affect others. Only the bucket-name registry is global (low write rate, strongly consistent, cached everywhere). Replication is async and per bucket. Small initial footprint uses the same code but checks that enough racks and nodes exist per AZ for "one fragment per failure domain" — if not, launches with replication and transcodes later.

Principal Approach: Codifies regional independence as an architectural rule ("no cross-region calls on the data path") and tests it with region-isolation game days. Plans the capacity ramp and the hardware cost curve, and makes sure the global control plane (bucket names, IAM) has its own availability design, since it's the only shared fate.

Key Takeaway: "Regions share only what they must, and what they share must fail gracefully."

What clears the Staff bar:

• Independent regional planes; minimal global state
• Durability honored even at small launch scale
• Tests isolation rather than assuming it

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Split blob storage into a metadata plane and a data plane and justify their different consistency models
• Walk the PUT path and explain why data-first, metadata-last makes crashes leave only garbage
• Compute durability for replication and erasure coding, and explain why repair time is the main lever
• Choose a code whose parity covers the largest per-AZ fragment count
• Explain range partitioning for LIST, why monotonic keys defeat split-on-load, and who fixes it
• Design multipart upload, soft delete, delayed GC, and orphan cleanup
• Name the correlated and software-caused failures that dominate real data loss
• Price redundancy and tiering choices at three scales and name the one-way doors

The Bar for This Question#

Mid-level (L4): Designs an upload service that stores files on servers with a metadata table. Mentions replication. Struggles with large uploads and listing.

Senior (L5): Separate metadata DB and storage nodes, 3× replication across AZs, multipart upload, CDN for reads. A system that works — but durability is asserted, LIST is an afterthought, deletes are immediate, and cost is 1.8× what it needs to be.

Staff+ (L6): Two planes with an explicit commit protocol; durability computed with repair time; zone-aware EC with a size-based hybrid; range-partitioned strongly consistent metadata with hot-prefix handling; soft deletes, delayed GC with a kill switch, scrubbing SLOs; each failure mode with a metric and owner. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Eleven Nines Is a Marketing Number — the Real Risk Is Software"#

The Staff position: Spend durability engineering on delete safety, GC paranoia, staged deploys, and audits — not on adding parity.

Why this matters in interviews: Candidates who say "more replicas" to every durability question signal they've never seen a real data-loss incident.

10.2 "Eventual Consistency Was Never Cheaper — It Was Just Billed to Someone Else"#

The Staff position: Pay for consistency once in the metadata plane.

Why this matters in interviews: It reframes a "performance vs consistency" debate as "who pays."

10.3 "Repair Speed Matters More Than Parity Count"#

The Staff position: Declustered placement and prioritized repair before wider codes.

Why this matters in interviews: It shows you understand the durability model rather than memorizing codes.

10.4 "Most Blob Storage Cost Is Garbage"#

The Staff position: Defaults (abort multipart after 7 days, noncurrent-version expiration, auto-tier among instant tiers) and chargeback beat any storage-engine optimization.

Why this matters in interviews: Cost questions reward governance answers, not compression answers.

10.5 "Don't Offer Rename"#

The Staff position: Flat namespace, no rename; offer hierarchical namespaces as a separate product only when demand justifies a second engine.

Why this matters in interviews: It shows you'll decline a feature to protect the core design.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer designs a durable, consistent object store. A Principal engineer sees that object storage is the organization's system of record by accident: every team dumps data into it, few write lifecycle rules, nobody owns cross-team retention, and the bill grows 40% a year while half the bytes are never read. The durability problem is solved; the governance problem isn't. The L7 question is how to make the org's storage spend and data-protection posture correct by default — and which decisions (namespace, consistency contract, key formats, vendor) will be impossible to reverse once 200 services depend on them.

The Org-Level Fault Line#

One storage platform with enforced defaults vs team-owned buckets with full autonomy.

The L7 default: The paved road. Every bucket is created through a platform module with versioning, encryption, lifecycle defaults, an owning team tag, and a data classification. Chargeback is per team per month. Quarterly audits flag unowned and non-compliant buckets.

🧭 Principal Move: "I'm not trying to make storage cheaper by engineering the engine — I'm making it cheaper by making every byte have an owner and a lifecycle. That's where the 30–40% is."

Cost Model#

Assumptions: self-hosted at scale; ~$6/TB-month for raw HDD capacity all-in (hardware amortized over 5 years, power, space, network); ~1.67× overhead for zone-aware EC; metadata and front-end fleet ~20–30% of data-plane cost; fully loaded engineer ~$25K/month. Public-cloud standard storage list price for comparison is on the order of ~$21–23/TB-month.

The number that matters to executives: below roughly tens of petabytes, building your own object store costs more than buying once people are counted. Above hundreds of petabytes, with steady growth and low egress needs, owning can save ~30–50% — but only with a team that treats durability as a permanent product.

The 3-Year Evolution Path#

Triggers: Year 1 when storage becomes a top-3 infra cost; Year 2 when the first tenant needs DR or compliance retention; Year 3 when a second region launches or a drive generation reaches end of life.

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Object Storage Data Protection & Cost Standard v1

Scope: All buckets holding production or customer data.

MUST:

• Create buckets through the platform module; every bucket has an owning team, a data classification, and a cost center tag.
• Enable versioning and encryption at rest on production buckets; noncurrent versions expire after a class-defined period (default 30 days).
• Abort incomplete multipart uploads after 7 days.
• Separate deletion privileges from write privileges for production buckets; bulk deletes above 10K objects/hour trigger an alert to the owner.
• Tier-0 data (customer records, financial ledgers) has a cross-account backup with object lock.

SHOULD:

• Use lifecycle rules to move data not read in 90 days to infrequent-access tiers.
• Use hashed or high-cardinality key prefixes for write rates above 1K/s.

Exceptions: Filed with the storage platform team; approved by platform lead and the data owner's director; reviewed every 6 months.

Success metrics: 100% of buckets owned; < 5% of bytes in incomplete uploads or orphaned versions; storage cost growth ≤ data growth; zero unrecoverable deletions of Tier-0 data.

What I'd Tell the VP#

Our storage is extremely unlikely to lose data because of hardware; the realistic risks are our own software bugs and accidental or malicious deletions, and our defaults today don't protect against those. I'm proposing a standard that turns on versioning and delete protection for production data and gives every bucket an owner. The same change attacks cost: today roughly a third of what we pay for is data nobody reads or has forgotten about, and chargeback plus lifecycle defaults should cut storage spend growth by 25–35% within a year. It needs about three engineers for two quarters. We should keep buying storage from our cloud provider until we're well past a hundred petabytes; building our own engine is not where the savings are.

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll add versioning" — right mechanism, no plan to make it the default across 2,000 buckets.
• "Erasure coding saves 44%" — correct math, no decision about whether the org should run the engine at all.
• "Storage team owns durability" — ignores that deletes, lifecycle, and retention are owned by hundreds of tenant teams.

Appendices

Appendix A: Durability Math in Depth#

A.1 The Model#

For a stripe of n fragments tolerating m losses, with per-disk failure rate λ (per year) and repair window T (years):

P_loss_per_year ≈ n·λ · C(n−1, m) · (λT)^m

Read it as: rate at which some fragment fails (n·λ), times the probability that m of the remaining n−1 also fail before repair completes.

(λ = 0.015/yr. Order-of-magnitude only; the model assumes independent failures and exponential lifetimes.)

A.2 What the Model Omits#

A.3 Copysets — Per-Object vs Fleet Loss#

With random placement across thousands of disks, almost any combination of m+1 simultaneous disk failures holds all surviving fragments of some stripe — so fleet-level "we lost something" events become frequent even when per-object probability is tiny. Restricting placement to a limited number of disk groups (copysets, Cidon et al., 2013) makes loss events rarer but larger. Choose deliberately: many small losses (random placement) or rare larger ones (copysets). Most object stores prefer fewer loss events, because each one is an incident with customer notification.

A.4 Repair Bandwidth#

bytes_read_per_disk_loss = disk_capacity × k        (RS: read k fragments per rebuilt fragment)
LRC single-loss read      = disk_capacity × group_size

24 TB disk, RS(9,6):  ~216 TB read cluster-wide
Declustered over 2,000 disks at 50 MB/s each for repair: ~100 GB/s -> ~36 min

Appendix B: Metadata Data Model#

Table object_versions  -- range-partitioned by (bucket_id, key)
  PK: (bucket_id, key, version_id DESC)
  cols: size, etag, checksum_crc64, storage_class, is_delete_marker,
        manifest (inline for <= 8 chunks, else pointer), created_at, kms_key_ref

Table uploads          -- partitioned by (bucket_id, key, upload_id)
  parts: map<part_no, {etag, size, chunk_refs}>, initiated_at

Table chunks           -- partitioned by chunk_id (hash)
  stripe_id, offset, length, refcount_hint

Table stripes          -- partitioned by stripe_id (hash)
  code, fragments[ {node_id, extent_id, offset, crc} ], sealed_at

Byte lifecycle — the states that the commit protocol and GC must respect:

Version IDs are time-ordered (e.g., hybrid logical clock + node ID) so "latest" is the first row in the key's range.

Appendix C: Coordination Mechanisms#

Quick comparison: A single global metadata database doesn't scale past ~10⁹–10¹⁰ rows comfortably; hash-sharded KV breaks LIST; range-partitioned consensus groups are the standard answer (the same shape as Spanner, CockroachDB, or TiKV).

Appendix D: API Contract and Client Behavior#

Appendix E: Observability#

E.1 Core Metrics#

Durability:  storage.stripes_by_surviving_fragments, repair.queue_age_p99,
             scrub.cycle_age_max, scrub.checksum_failures, gc.marked_bytes_ratio
Metadata:    metadata.partition_qps, metadata.commit_latency_p99, metadata.split_rate
API:         api.latency_p99{op}, api.error_rate{op}, api.throttled_total{tenant}
Cost:        storage.bytes{tenant,class}, multipart.incomplete_bytes, versions.noncurrent_bytes
Data path:   storage.node_egress_gbps, get.reconstructed_reads_ratio

E.2 Critical Alerts#

E.3 Control Plane vs Data Plane#

Control plane: bucket creation, policy, lifecycle config, placement maps, partition management — may be briefly unavailable. Data plane: GET/PUT/LIST on existing buckets — must keep working with cached control-plane state. A control-plane outage should block new bucket creation, not reads.

E.4 Debugging "Is Any Data at Risk Right Now?"#

1. storage.stripes_by_surviving_fragments histogram — anything at minimum?
2. Repair queue age and throughput — is repair keeping up?
3. Failure-domain view — are failures clustered (rack, model, AZ)?
4. GC and scrub health — both running, both within SLO?

Appendix F: Scale Evolution#

F.1 What Works at Each Scale#

F.2 Multi-Region Path#

Independent regional planes; a global bucket registry and IAM; async per-bucket replication with lag SLOs; no cross-region calls on the data path (Deep Dive 5).

F.3 What You Don't Build on Day One#

• Hierarchical namespace
• Archive tiers with hours-long retrieval
• Cross-region replication
• LRC or custom codes (start with RS)
• Object lock / compliance modes (until a tenant needs it)

Appendix G: Multi-Tenancy, Fairness, and Cost#

🧭 Principal Insight: "Price what costs us money. If LIST and small objects are what load the metadata plane, bill for them — otherwise we subsidize the workloads that hurt us most."

Related reading: Large Blobs, File Sync, CDN & Edge Caching, Database Sharding, Consistency Models, and Sharding.