Technologies referenced in this case study: PostgreSQL · DynamoDB · Apache Kafka · Redis · Cassandra · Kubernetes

Related: Object Storage · CDN · Large File Uploads & Delivery · Video Streaming · Cloud File Sync · News Feed · Idempotency & Exactly-Once · Transactional Outbox · Security Fundamentals · Key Management Service

Reading Guide#

Organized for interview use first, reference second. Read front-to-back once, then return to the fault lines and incidents that match your weak spots.

Without resumable uploads and an upload state machine:
t=0:        User starts uploading a 40-photo album (160 MB) on a commuter train.
t=+45s:     Tunnel. Connection drops at photo 23, 60% through a 4 MB single PUT.
t=+46s:     App retries photos 1–40 from scratch. Photos 1–22 now exist twice.
t=+2min:    Feed shows the album with 22 duplicates and 3 grey boxes
            (metadata committed before thumbnails existed).
t=+1 day:   Support tickets: "my album is broken". 18 TB/month of orphaned bytes
            from abandoned uploads nobody is cleaning up.

With resumable chunked uploads, client upload IDs and a state machine:
t=0:        App creates 40 upload sessions with client-generated upload IDs.
t=+45s:     Tunnel. Photo 23 has 2 of 3 chunks acknowledged.
t=+3min:    Signal returns. App asks the server which chunks exist, sends chunk 3.
t=+3min:    Retried "create" calls return the existing sessions — zero duplicates.
t=+3min:    Each photo appears in the album only when its thumbnails are READY;
            until then the uploader sees a local preview, followers see nothing.
t=+7 days:  Lifecycle rule aborts the 0.4% of sessions that never finished.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What the Interviewer Is Scoring#

Photo upload is not a file-transfer question. Everyone can draw a presigned URL and a bucket.

It is a lifecycle and cost question that tests:

• Whether an upload is an idempotent, resumable session rather than a single request that either works or doesn't
• Whether you define exactly when a photo becomes visible — and never show a reference to bytes or derivatives that don't exist yet
• Whether you treat every uploaded file as hostile input to a decoder running on your servers
• Whether you price storage over years, not days — because bytes are forever and the bill compounds
• Whether "delete" means the bytes are gone everywhere, including caches, derivatives and backups

The key insight: A photo is written once, read thousands of times in its first week and almost never after a year — and kept for a decade. The design follows that curve: make the write path resumable and the first week fast, then move bytes down tiers as reads fall off, and make deletion a first-class pipeline instead of an afterthought. Senior candidates design the upload; Staff candidates design the photo's whole life.

One Question, Three Levels#

Positions to Commit To#

Which Problem Are We Solving?#

Three intents produce three different systems. Name them, then commit.

🎯 Staff Move: "I'll design social photo sharing — upload from phones, visible in feeds within seconds, heavy reads early, kept for years. I'll borrow two things from the backup world: originals stored at full resolution, and a real tiering policy, because the storage curve is what decides cost. If this were a marketplace, moderation would sit between upload and visibility, which changes the state machine but not the storage."

Where the Design Splits#

How Real Companies Built It#

Why this section belongs here: Two published papers describe how one of the largest photo services stored hot and warm photos differently, and the most widely used object store documents the upload primitives every design leans on. Naming them shows you know where the real constraints come from.

Facebook Haystack — One Disk Operation per Photo Read#

Facebook's OSDI 2010 paper describes Haystack, built because storing each photo as a file on NFS-mounted appliances spent most disk I/O on filesystem metadata: the paper reports more than 10 disk operations to read a single image in directories with thousands of files, and about 3 even after tuning. At the time Facebook stored over 260 billion images (four sizes generated per uploaded photo) — more than 20 PB — with users uploading about one billion new photos (~60 TB) per week. Haystack packs many photos ("needles") into large append-only volume files and keeps the per-photo metadata small enough to hold in memory, aiming for at most one disk operation per read. The paper also explains why a CDN alone was not enough: the long tail of requests for less-popular photos misses the CDN, so the storage tier must serve it efficiently (Haystack paper, OSDI 2010).

Staff insight: The bottleneck at scale was not bandwidth or capacity — it was metadata I/O per object. In an interview, say: "Photos are small, immutable and numerous. I'd store them packed in large objects or rely on an object store that does, and keep the per-photo index in memory or a fast KV store — the long tail misses the CDN, so the origin has to be cheap per read."

Facebook f4 — Warm Storage and Deletion by Key#

Facebook's OSDI 2014 paper describes f4, a warm BLOB store added next to Haystack once it measured that requests and deletes fall sharply with age. Haystack's triple replication with RAID-6 gave an effective replication factor of 3.6; f4 uses Reed-Solomon(10,4) within a datacenter plus XOR coding across geographic regions, bringing the effective replication factor to 2.8 and then 2.1. The paper reports that for most BLOB types about a month was a safe boundary between hot and warm; photos used a three-month threshold and profile photos were never moved. Each BLOB in f4 is encrypted with a per-BLOB key stored in an external database, and deleting the key logically deletes the BLOB — so f4 does not need to reclaim space quickly after deletes (f4 paper, OSDI 2014).

Staff insight: Two interview-grade lessons in one system: tier by measured access curve (three months for photos, not a guess), and make deletion a key operation instead of a byte operation. Say: "Deleting the key is the delete. Compaction can happen whenever it's cheap."

Amazon S3 — Multipart Uploads, Presigned URLs and Abandoned Parts#

Amazon S3's documentation recommends multipart upload once objects reach about 100 MB and specifies parts of 5 MiB to 5 GiB (no minimum for the last part), up to 10,000 parts, and a 48.8 TiB maximum object size (S3 multipart limits). Presigned URLs can be generated with SDKs for up to 7 days, expire early if the signing credentials expire, can be used multiple times until expiry, and an upload to a presigned key replaces any existing object at that key (S3 presigned URLs). S3 recommends a lifecycle rule with AbortIncompleteMultipartUpload so uploads that never complete have their parts deleted after a set number of days (S3 lifecycle for incomplete uploads).

Staff insight: Three operational facts hide in those docs. A presigned URL is a reusable bearer token for its lifetime, so sign minutes, not days, and never sign a key a client chose. A PUT overwrites, so upload to a server-chosen staging key and promote it. And abandoned parts are billed storage until a lifecycle rule aborts them — say so before the interviewer asks.

Follow-Ups to Expect#

System Architecture Overview#

Reading the diagram: The API is a control plane: it creates idempotent upload sessions and signs part URLs, but never carries bytes. Clients upload parts straight to a staging bucket. Completion emits an event; sandboxed processors validate the file, strip location metadata, write the canonical original and a fixed set of derivatives into the hot tier, then flip the metadata row to READY. Viewers fetch immutable derivative URLs through the CDN; rare sizes are rendered on demand. Lifecycle rules move originals to erasure-coded warm storage after ~90 days. The deletion pipeline tombstones, purges and finally destroys. The metric that tells you uploads are healthy is media.pending_age_p99 — how long photos sit between "bytes arrived" and "visible".

One-Minute Recap#

Numbers to Bring#

Interview Walkthrough

The most common mistake: Candidates spend 15 minutes on the presigned URL flow and the thumbnail Lambda, then run out of time before the interviewer asks the questions that decide the level: "What does a follower see before the thumbnail exists?", "What does storage cost in year five?" and "The user deletes the photo — where does it still exist?" Compress the happy path to ~8 minutes and spend the rest on the session and visibility contract, processing as untrusted input, tiering and deletion.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional scope in one breath:

"Users upload photos from phones and web, see them in their own grid immediately, and share them to feeds where followers see them within seconds. We store originals at full resolution, serve several display sizes through a CDN, and support delete. Uploads come from mobile networks that drop constantly."

Then the non-functional requirements, which is where the design lives:

"Four constraints drive everything. One: uploads must survive dropped connections without duplicates. Two: nothing references a photo before it's renderable — no grey boxes. Three: every upload is untrusted input to code running on our servers. Four: storage is forever and compounds, so cost per PB-month by tier is a design input. I'll assume 50 million uploads a day, averaging 2.5 MB, peaking around 3× average — about 1,700 uploads a second at peak and roughly 125 TB of new originals a day — with reads around 100× writes, mostly in the first week."

Then name the underspecified parts:

"I'd confirm: do we keep originals at full resolution? Are photos public, followers-only or private? Is there a moderation step before visibility? What's the deletion promise? I'll assume originals kept, mixed visibility with signed URLs for private photos, async moderation after visibility for social posts, and gone from live systems within 30 days of delete."

🎯 Staff Move: Saying "storage is forever and compounds" in the first two minutes reframes the problem from "move bytes" to "own a photo's whole life". It earns you the right to spend time on tiering and deletion later.

Phase 2: Core Entities & API (1–2 minutes)#

Name the nouns in 30 seconds:

• UploadSession: upload_id (client-generated UUID), owner_id, declared_size, declared_type, part_size, parts_received[], staging_key (server-chosen), state (OPEN / COMPLETE / ABORTED), expires_at
• Photo: photo_id, owner_id, upload_id (unique), state (PENDING / READY / FAILED / QUARANTINED / DELETED), content_sha256, width, height, taken_at, visibility, blob_ref, derivative_version
• Blob: blob_id, sha256, size, tier, locations, owner_scope (per-user dedup), refcount, key_id
• Derivative: photo_id, spec (e.g. thumb_256, display_1080), blob_id, format (WebP/AVIF/JPEG)

Client-facing API:

POST   /v1/uploads                    { upload_id, size, type, sha256? }     → session, part_size, signed part URLs (first 4)
GET    /v1/uploads/{upload_id}        → parts_received[], state              (resume)
POST   /v1/uploads/{upload_id}/parts  { part_numbers[] }                     → more signed part URLs (lazy, 15-min expiry)
POST   /v1/uploads/{upload_id}/complete { parts[{n, etag}], caption?, visibility } → photo_id, state=PENDING
GET    /v1/photos/{photo_id}          → state, derivative URLs when READY
DELETE /v1/photos/{photo_id}          → state=DELETED (tombstone), restorable 30 days

🎯 Staff Move: "The client generates the upload ID before the first request. Create, part uploads and complete are all idempotent on it, so a retry after a timeout returns the same session and the same photo ID — not a second photo. That's the one decision that makes mobile uploads correct."

Phase 3: High-Level Architecture (≤5 minutes)#

Draw at most eight boxes:

Walk one photo in 90 seconds:

1. The app generates upload_id, computes SHA-256 while reading the file, and calls create. The API writes an OPEN session and returns signed URLs for the first parts, each valid 15 minutes, each bound to a server-chosen staging key.
2. The app PUTs 8 MB parts in parallel (2–4 at a time on cellular), retrying each with backoff. On reconnect it asks which parts landed and sends only the missing ones.
3. Complete: the API asks storage to assemble the parts, verifies size and checksum, creates the Photo row as PENDING in the same transaction that closes the session, and emits a media event via an outbox.
4. A processor pulls the event, decodes the image inside a sandbox with pixel and time limits, strips GPS and other sensitive EXIF, normalizes orientation, writes the canonical original and five derivatives under content-addressed keys.
5. The processor flips the photo PENDING → READY with a conditional update and publishes photo.ready. The feed fan-out starts from that event, not from the upload.
6. Viewers request https://cdn…/d/{sha256-of-derivative}.webp; the CDN caches it for a year because the URL never changes for different bytes.

🎯 Staff Move: Say out loud: "The feed fans out on photo.ready, not on upload complete. That single choice is why followers never see a broken image." You've now spent ~8 minutes.

Phase 4: Transition to Depth (1 minute)#

"That's the happy path, and it's the Senior-level design. What makes this hard is that mobile uploads fail constantly, every file is a potential exploit against our decoders, storage grows by tens of petabytes a year, and delete has to be true everywhere. I'd like to go deep on resumable sessions and the visibility contract, the processing pipeline, storage tiering and cost, and deletion and privacy. Where would you like to start?"

If no preference: start with sessions and visibility. It's the part where Senior designs quietly break.

Phase 5: Deep Dives (25–30 minutes)#

For each: state the tradeoff → commit → quantify → name who pays.

Deep dive 1: Resumable sessions and the visibility contract (7–8 min)

"A session is a server-side record keyed by the client's upload ID. Parts are 8 MB — a 4 MB photo is one part, a 60 MB burst-mode file is eight. Part URLs are signed lazily, a few at a time, for 15 minutes, so a leaked URL is useless quickly. Completion is idempotent: if the session is already COMPLETE, return the existing photo ID. The Photo row starts PENDING. Only READY photos are visible to anyone but the owner; the owner's app renders the local file it already has."

Quantify: "At 1,700 uploads a second, sessions are ~150M rows a day with a 7-day TTL — about a billion live rows, small, in a KV store keyed by upload_id. 1–5% never complete; their parts are deleted by a lifecycle abort rule at 7 days. At 2% abandonment and an average 3 MB of parts, that's ~3 TB a day that would otherwise accumulate forever."

Who pays: "The client SDK team carries the complexity of chunking and resume — that's correct, because it's the only place that knows what's on disk."

Deep dive 2: The processing pipeline as an untrusted-input boundary (6–7 min)

"Image decoders are large C libraries parsing attacker-controlled bytes. Processors run in a sandbox — no network except to the blob store, a read-only filesystem, a seccomp profile — with limits: reject anything over 100 megapixels before decoding pixels, 50 MB input, 10 seconds CPU. We sniff the real format from magic bytes, never trust the extension, and re-encode every output so nothing from the original byte stream is served raw. EXIF GPS and device serials are stripped from derivatives by default; the original keeps them encrypted and owner-only."

Quantify: "A 12 MP decode plus five resizes and WebP encodes is ~300–800 ms of CPU. At 1,700 photos a second peak, that's ~1,000 cores busy, so ~1,500 cores with headroom — autoscaled on queue age, not depth."

Deep dive 3: Storage tiering and cost (5–6 min)

"Derivatives are ~15% of bytes and get almost all reads — they stay hot. Originals are read for edits, downloads and re-renders, mostly in the first weeks. Lifecycle moves originals older than 90 days to erasure-coded warm storage at ~1.4× overhead instead of ~3×. Library-style originals older than a year that haven't been opened can go to a cold class with a restore path that takes minutes to hours, and the UI says so."

Quantify: "125 TB a day is ~45 PB of originals a year. At illustrative prices — $20 per TB-month hot, $10 warm, $2 cold — keeping everything hot costs ~$900K a month by the end of year one and ~$4.5M a month at year five; hot-for-90-days-then-warm brings year five to roughly $2.4M a month, and a cold tier for unopened library originals takes it lower." (Run your own assumptions through the cost estimator.)

Deep dive 4: Delivery (3–4 min)

"Derivative URLs are content-addressed — the hash of the derivative bytes is in the path — so they're immutable and cached for a year. Public photos are plain CDN objects behind an origin shield. Private photos use short-lived signed URLs or signed cookies, scoped to a path prefix. A re-render with a new encoder produces new hashes and new URLs; old URLs age out of caches."

Deep dive 5: Deletion and privacy (4–5 min)

"Delete flips the row to DELETED — every API stops returning it within a second — and enqueues a CDN purge for its derivative paths. After a 30-day recycle bin, a deletion job removes the original and derivatives and decrements refcounts on deduplicated blobs. Backups can't be rewritten per photo, so originals are encrypted under per-user keys from the KMS; account erasure destroys the user's key, which makes every backup copy unreadable. Every deletion request has an SLA timer and an audit record."

Phase 6: Wrap-Up (2–3 minutes)#

"The core idea: a photo is written once, read heavily for a week and kept for a decade, so the design follows that curve. Idempotent resumable sessions for the write; a READY state that gates visibility; a sandboxed pipeline because uploads are hostile input; immutable URLs for a CDN-first read path; tiering that tracks the access curve; and a deletion pipeline that reaches the CDN, the derivatives and the backups."

The evolution closer:

"What I'd build later: on-demand rendering for the long tail of sizes and new formats like AVIF; client-side dedup for the backup product; perceptual-hash near-duplicate detection; regional storage for data residency. What I'd not build early: our own blob store. Renting object storage is right until the bill is large enough to fund a storage team — and that's a multi-year decision."

🎯 Staff Move: End on who owns "gone". "The media platform owns the deletion SLA end to end — tombstone, purge, destroy, shred — with per-request evidence. Feed and search consume photo.deleted and own removing their own copies within the same SLA."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Photo storage is the interview where time is the hidden axis. The upload lasts seconds; the processing lasts seconds; the photo lasts ten years, through three storage tiers, two encoder generations, a data-residency law and a deletion request. A Senior design is correct at t=0. A Staff design is correct at t=+5 years — when the storage bill is the largest line item, when a decoder CVE forces reprocessing of a billion images, and when a regulator asks to see proof that one specific user's photos are gone.

It also has a deceptive read pattern. Reads look enormous — 100× writes — but the CDN absorbs 95% of them, and the ones that reach origin are disproportionately the long tail: old photos nobody has cached. The expensive system is not the one serving viral photos; it's the one storing and occasionally serving the 99% of photos that are almost never viewed.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"A user uploaded a photo three years ago, shared it publicly, and today deletes their account. Walk me through every place a copy of that photo exists right now, and when each one goes away."

A candidate who lists the metadata row (tombstoned in under a second), the CDN edges and origin shield (purged within minutes; immutable URLs mean nothing else invalidates them), the hot derivatives and warm-tier original (deleted after the recycle window), deduplicated blobs (refcount decremented, bytes freed only at zero, and only within that user's scope), replicas and erasure-coded fragments (deleted with the object), backups (unreadable once the per-user key is destroyed), downstream copies in feed caches and search indexes (removed on photo.deleted within the same SLA), and copies already downloaded by other people (outside our control, and said so honestly) — that candidate has run a media platform. A candidate who says "we delete it from S3" has built an upload form.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Social photo sharing → time-to-visible and read fan-out

• Constraint: seconds from upload to followers seeing it; huge early reads; viral hot objects
• Strategy: resumable upload, eager small derivatives, READY gate, CDN-first delivery, age-based tiering
• Failure mode: grey boxes, duplicates on retry, origin melt on viral photos, slow first view
• Who pays for imperfection: the uploader (embarrassment), followers (broken feed), the CDN bill

Personal library / backup → durability and cost over decades

• Constraint: every original kept at full resolution; background bulk sync of tens of thousands of photos; rare reads of old photos
• Strategy: client-side hashing, per-user dedup before upload, aggressive cold tiering, explicit restore UX
• Failure mode: silent loss, duplicate libraries after reinstall, surprise restore latency on old photos
• Who pays: users (lost memories are not refundable), finance (cold-storage retrieval fees)

Marketplace / UGC listings → moderation and takedown

• Constraint: nothing visible before moderation; uniform derivative specs; legal takedown in minutes
• Strategy: upload → moderation queue → publish state; strict crops; purge-on-takedown
• Failure mode: unmoderated image visible for minutes; takedown leaves cached copies
• Who pays: trust and safety, legal

2.2 When NOT to Build a Photo Platform#

• Images are incidental to your product (avatars, a few attachments). Use managed object storage with presigned uploads and a hosted image-transformation CDN feature. A pipeline team is overkill below a few million images.
• Your files are large and streamed (video). Use the video streaming design: segmenting, adaptive bitrate, packaging. The upload half is similar; the processing and delivery halves are not.
• You need file sync semantics (folders, edits, conflicts across devices). That's cloud file sync: versioning and conflict resolution dominate.
• You'd build your own blob store at < ~10 PB. Rented object storage wins on durability engineering you won't match with a small team; revisit only when the bill funds a storage organization.

🎯 Staff Insight: "I'd only build the pipeline and the lifecycle. The blob store, the CDN and the image codecs are bought or open-source. The parts that are ours are the visibility contract, the cost policy and the deletion promise."

2.3 What the Interviewer Leaves Underspecified#

Interviewers deliberately omit:

• Whether originals are kept — many products only need a 2048-pixel version; keeping originals at full resolution can double storage
• Visibility model — public, followers-only or private changes whether URLs are signed and whether CDN caching is per user
• Moderation timing — before visibility (marketplace) or after (social), which changes the state machine
• Deletion promise — "deleted" in the UI, or provably erased from backups by a deadline
• Size distribution — a mix of 3 MB phone photos and 80 MB RAW files needs a chunked path either way
• Retention of location data — EXIF GPS in a public photo reveals a home address

Staff engineers surface these and commit. Senior engineers assume them away and get surprised by the follow-up.

2.4 Precise Terminology#

🎯 Staff Insight: If the interviewer says "just store the photos", ask: "For how long, at what resolution, and what does delete promise? Those three answers set 80% of the cost and most of the compliance risk."

3. Where the Design Splits#

Every photo decision has a technical side (chunk sizes, codecs, erasure codes) and an organizational side (what users are promised, who owns erasure, who pays for storage growth). Interviewers grade the second side.

3.1 Fault Line 1: Upload Path — Proxy vs Presigned, Single vs Resumable#

The tension: Proxying bytes through your API gives full control and inline inspection, but every slow mobile client pins an API connection for the full upload. Presigned direct upload offloads ingest to storage but enforces policy only through what you sign. Single-request uploads are simple; resumable sessions survive the networks your users actually have.

Staff default: "Presigned, resumable chunked sessions. The client generates the upload ID, part size is 8 MB (4 MB on 2G-class networks), part URLs are signed lazily for 15 minutes each and bound to a server-chosen staging key, content type and part length. Completion validates total size and the client-supplied SHA-256. A lifecycle rule aborts sessions after 7 days."

When to deviate:

• Inline scanning is mandatory before bytes land anywhere (some regulated contexts): a dedicated upload edge fleet that streams to storage, not the general API.
• Tiny images only (avatars < 1 MB): a single presigned PUT is fine; resumability buys little.

🧭 Principal Move: "Every product that ingests media — posts, messaging, support attachments, marketplace — should use one upload SDK and one session service. Otherwise five teams build five resumable protocols, and three of them don't abort orphaned parts."

❌ Common L5 Trap: "The client uploads to a presigned URL for photos/{user_id}/{filename}." The client now chooses part of the final key, a retry with the same name overwrites a different photo, a URL valid for a day is a reusable bearer token for that path, and a drop at 90% restarts the whole file.

3.2 Fault Line 2: Derivatives — Eager vs On-Demand#

The tension: Rendering every size at upload makes first view instant but stores sizes nobody requests and makes adding a format a billion-image backfill. Rendering on demand stores nothing extra but puts a resize fleet behind every CDN miss — and a viral photo's first minute behind a stampede.

Staff default: "Hybrid. Five eager derivatives — a 256-pixel thumbnail, 640, 1080, 1440 and a blurred placeholder of a few hundred bytes — in WebP with a JPEG fallback. They add ~15% to stored bytes and cover almost every view. Anything else renders on demand from the original, with request coalescing at the origin shield and the resizer, so a viral photo triggers one render per size, not ten thousand."

When to deviate:

• Marketplace with strict specs: eager everything — the set is small and fixed, and moderation reviews the derivatives.
• Archive/library product with rare views: on-demand for almost everything beyond the thumbnail.

🎯 Staff Insight: "The eager set is a product decision measured in bytes. I'd check request logs: if 97% of views hit four sizes, those four are eager and the rest aren't worth storing for a billion photos."

❌ Common L5 Trap: "We'll generate twelve sizes in three formats at upload, so every device gets a perfect fit." Thirty-six derivatives per photo multiply object count by 36 and add a backfill of billions of objects every time product wants a new size or format.

3.3 Fault Line 3: Dedup — Global vs Per-User vs None#

The tension: Content-hash dedup saves storage — memes and forwarded images are uploaded millions of times. But global dedup lets one user learn whether another user has a specific file, couples deletions across users, and makes per-user encryption impossible.

Staff default: "Per-user dedup on originals. The client sends the SHA-256 at session create; if that user already has a READY photo with the same hash, we skip the upload and create a new photo row pointing at the same blob, refcount + 1. No cross-user dedup on private originals: it would tell an attacker whether a target has a specific file, and it would make per-user crypto-shredding impossible."

When to deviate:

• Public, widely forwarded content (stickers, memes in messaging): global dedup scoped to content already public, where existence isn't a secret.
• Backup product at huge scale: per-user dedup is the large win — reinstalls and re-syncs are the common duplicate.

🧭 Principal Move: "Dedup scope and encryption scope must be the same. If I want to shred a user's data by destroying one key, no blob can be shared outside that user. I'd write that down as a rule before someone 'optimizes' storage with global dedup."

❌ Common L5 Trap: "Hash every upload and store each unique file once across the whole platform — huge savings." Then "skip upload, already have it" responses reveal that someone, somewhere, uploaded a given file, and deleting a photo needs a cross-tenant refcount that's one bug away from deleting another user's memories.

3.4 Fault Line 4: Tiering — How Fast Bytes Move Down#

The tension: Hot replicated storage serves anything fast and costs the most. Warm erasure-coded storage halves the cost but small reads and repairs are slower. Cold archive is an order of magnitude cheaper and takes minutes to hours to read, with retrieval fees. The access curve decides where the lines go.

Staff default: "Age-based, because it's predictable and the read curve for social photos is steep. Derivatives stay hot — they're ~15% of bytes and nearly all reads. Originals move to erasure-coded warm at 90 days. For the library product, originals unopened for a year go cold; the app shows the thumbnail instantly and labels full-resolution as 'restoring'. Tiering is lifecycle configuration in the storage layer, versioned and reviewed, not a cron job."

When to deviate:

• Profile photos and pinned content: never leave hot; they're read constantly regardless of age. (f4 made the same exception for profile photos.)
• Regulated retention: some content can't go to a class or region that violates residency — tier within the allowed region.

🧭 Principal Move: "Tier thresholds are a cost-versus-experience contract. I'd have product sign off on what 'restoring' feels like and finance sign off on retrieval budgets, then revisit the thresholds yearly from measured access curves — the way f4 picked three months for photos from data."

❌ Common L5 Trap: "Move everything older than 30 days to Glacier to save money." A user scrolls back to last summer and every full-resolution tap takes hours, retrieval fees spike when a "memories" feature resurfaces old photos to millions of users at once, and the savings vanish.

3.5 Fault Line 5: Deletion and Privacy — How "Gone" Is Gone#

The tension: Soft deletes are fast and reversible; real erasure has to chase copies across caches, derivatives, replicas, dedup references and backups that were designed to make data hard to lose. Privacy also starts at upload: location metadata and private-photo URLs leak more than users expect.

Staff default: "Delete is a pipeline with an SLA: tombstone within a second, CDN purge within 15 minutes, a 30-day recycle bin, then destruction of original and derivatives; downstream systems consume photo.deleted and must comply within the same SLA. Originals are encrypted under per-user data keys wrapped by the KMS, so account erasure destroys the key and every backup copy becomes unreadable. GPS is stripped from derivatives at processing time; private photos are served with signed cookies scoped to the owner's grant, valid for an hour."

When to deviate:

• Legal hold: a hold flag blocks destruction (not the tombstone) and is visible to the deletion pipeline and the audit trail.
• Abuse evidence: content removed for policy violations may be preserved under a separate, access-controlled retention regime set by legal.

🧭 Principal Move: "I'd make 'every system that copies media must consume photo.deleted' a platform rule with an audit. The media store can be perfect and we still fail an erasure request because a search index kept a thumbnail."

❌ Common L5 Trap: "Set the CDN TTL short so deletes propagate." Short TTLs crush cache hit rates for the 99.9% of photos that are never deleted. Long TTLs on immutable URLs plus explicit purge on delete gives you both.

4. When It Breaks#

4.1 The Retry Storm of Duplicates — Uploads Without Idempotency#

t=0:       A CDN-fronted API region degrades: create-session p99 goes 120ms → 9s.
t=+10s:    Mobile client timeout is 8s. Clients retry create with a NEW server-generated ID.
t=+1min:   Each slow request eventually succeeds AND its retry succeeds → 2–3 sessions per photo.
t=+5min:   Both sessions complete. 340K photos posted twice, some three times.
t=+20min:  Users delete duplicates by hand; feed shows double posts; support queue +12K.
t=+1 day:  Dedup cleanup job has to guess which duplicate the user meant to keep.

Detection: upload.sessions_per_unique_sha256{owner} > 1.05; photo.duplicate_posts_total; create-session latency vs client timeout.

Mitigation: server-side dedup of PENDING photos with the same owner and SHA-256 within 10 minutes; keep the earliest.

Prevention: client-generated upload_id, unique index on (owner_id, upload_id); create returns the existing session on conflict; client timeout tuned above server p99.9.

Owner: media platform (session service), client SDK team (ID generation and retry policy).

4.2 The Grey-Box Feed — Fan-Out Before READY#

t=0:       A deploy changes the processor's WebP encoder; 6% of photos fail encoding.
t=+0:      The old design fans out to feeds on upload complete, not on READY.
t=+2min:   Followers' feeds render 6% of new photos as grey placeholders.
t=+10min:  Clients cache the broken derivative state for the session.
t=+25min:  Rollback. Failed photos reprocessed. Clients still show grey until refresh.

Detection: media.pending_age_p99 > 30s; media.failed_total rate; client-reported image.load_error_rate.

Mitigation: roll back the encoder; reprocess FAILED photos from their stored originals; push a photo.ready update so clients refetch.

Prevention: fan-out only on photo.ready; canary processor deploys on 1% of traffic with automatic rollback on failure-rate delta > 0.5%; derivative validation (decode the output) before READY.

Owner: media platform (pipeline); feed team (consumes photo.ready, not upload events).

4.3 The Decompression Bomb#

t=0:       Attacker uploads a 180 KB PNG that declares 60,000 × 60,000 pixels.
t=+1s:     Processor allocates ~14 GB for the RGBA buffer. OOM-killed.
t=+1s:     Event redelivered (at-least-once). Next processor OOMs. Repeat.
t=+3min:   Attacker has uploaded 400 such files. 30% of processor pods crash-looping.
t=+5min:   media.pending_age_p99: 3s → 6 min for every user.

Detection: processor.oom_kills; queue.redelivery_count{event} > 3; pending age.

Mitigation: read the header and reject declared dimensions over 100 megapixels before allocating pixels; poison-message handling — after 3 failed attempts, mark QUARANTINED and stop redelivering.

Prevention: limits checked before decode (dimensions, frame count for animated formats, input size, CPU time); per-process memory caps; a per-user upload rate limit; redelivery cap with quarantine as the terminal state.

Owner: media platform (pipeline), security (decoder hardening and CVE response).

4.4 The Orphaned Parts Bill#

Month 0:   Resumable uploads launch. No AbortIncompleteMultipartUpload rule on staging.
Month 1:   2.3% of sessions abandoned (app killed, user cancels, network gone).
Month 6:   Staging bucket holds 610 TB of parts that never became objects.
           They don't appear in object listings; cost reports show "storage" growing.
Month 7:   Finance asks why storage grew 9% faster than uploads. Nobody owns staging.

Detection: staging.incomplete_upload_bytes; ratio of staging bytes to completed-object bytes; storage-class cost breakdown.

Mitigation: add a lifecycle rule to abort incomplete uploads after 7 days; one-time sweep of existing parts.

Prevention: the abort rule ships with the bucket definition, reviewed in infrastructure code; a dashboard of staging bytes with an owner.

Owner: media platform (staging bucket is ours, not "the storage team's").

4.5 The Delete That Didn't — Cached Derivative Outlives the Photo#

t=0:       User deletes a photo after realizing it shows their house number.
t=+1s:     API returns 404 for the photo. User relieved.
t=+1h:     A friend opens an old share link; the CDN serves the cached 1080 derivative.
t=+3 days: User reports it. Purge job had failed on a CDN API rate limit and never retried.

Detection: delete.purge_pending_age_p99; a canary that deletes a test photo every 5 minutes and probes edge URLs for 404.

Mitigation: retry purges with backoff and a dead-letter queue that pages after 1 hour; purge by surrogate key/tag (all derivatives of a photo) rather than per URL.

Prevention: deletion SLA tracked per request; purge as a durable workflow step, not a fire-and-forget call; signed URLs for non-public photos so a stale cache entry still requires a valid signature.

Owner: media platform (deletion pipeline).

4.6 The Viral Origin Melt#

A celebrity's photo is shared to 80 million followers within minutes. Eager derivatives are cached fine — but a new client version requests a 2160-pixel size that isn't in the eager set. Thousands of edge POPs miss simultaneously; without coalescing, the resizer receives 40,000 identical render requests in a minute and falls over, taking every on-demand size down with it.

Detection: resizer.inflight_per_key max; cdn.origin_requests{photo} top-N; resizer error rate.

Mitigation: origin shield in front of the resizer, request coalescing per (photo, spec), and a degraded fallback that serves the nearest eager size.

Prevention: new client sizes added to the eager set (or pre-warmed) before the client ships; load test a single hot key at 50K requests/minute.

Owner: media platform; client teams own announcing new sizes.

4.7 Operational Reality Matrix#

🎯 Staff Insight: The pipeline's health metric is time-to-READY for new uploads, not queue depth. "Backfills and reprocessing can put millions of events on the queue legitimately. I'd run new uploads in their own priority lane and page on media.pending_age_p99 for that lane only."

5. Scorecard#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Deep S3 API knowledge ≠ upload design. Knowing every SigV4 header doesn't answer what happens on retry or when the photo becomes visible.
• Many derivative formats ≠ good processing. Twelve sizes in three codecs is a storage multiplier, not a feature.
• "We use a CDN" ≠ a delivery design. Without immutable URLs and a purge path, the CDN is either stale or useless.
• Encryption at rest ≠ deletion. Server-side encryption with one bucket key doesn't let you shred one user's backups.

6. The 45 Minutes, Phase by Phase#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• SigV4 internals — "short-lived, bound to key, length and type."
• Codec details — "WebP with JPEG fallback; AVIF later" is enough.
• Image ML (tagging, faces) — consumers of photo.ready, out of scope.
• Feed ranking — the feed consumes events; it's a different interview.
• Building a blob store — say when you would, in one sentence.

6.4 Follow-Up Questions to Expect#

1. "The connection drops at 80% of a 50 MB upload. Walk me through the next minute."
2. "When exactly can a follower see the photo, and what do they see before that?"
3. "How do you stop a malicious image from taking down the processing fleet?"
4. "Would you dedupe across users? Why or why not?"
5. "What does storage cost in year five, and which three changes cut it most?"
6. "The user deletes a photo. List every place a copy exists and when it goes away."
7. "A celebrity posts and a new client requests a size you don't pre-render. What happens?"

7. Practice Rounds#

Drill 1: The Opening#

Prompt: "Design photo upload and storage for a social app."

Drill 2: The Dropped Upload#

Prompt: "A user uploads a 60 MB burst-mode file on a train. The connection drops at 80%. What happens?"

Drill 3: Make It Concrete — Size Ingest, Processing and Storage#

Prompt: "Size the upload path, the processing fleet and year-one storage."

Drill 4: When Is the Photo Visible?#

Prompt: "A user posts a photo. When can their followers see it, and what do they see before that?"

Drill 5: The Hostile File#

Prompt: "How do you keep a malicious image from compromising or crashing your processing fleet?"

Drill 6: Dedup and Privacy#

Prompt: "Should we deduplicate identical photos across all users to save storage?"

Drill 7: Build vs Buy#

Prompt: "Should we build our own blob storage and image pipeline or use cloud services?"

Drill 8: Changing the Derivative Set Without an Outage#

Prompt: "Product wants AVIF derivatives for all photos, including the existing 40 billion. How do you ship it?"

Drill 9: The Cost of Storage#

Prompt: "Storage is now the biggest line on the infra bill. Cut it by 40% without losing photos."

Drill 10: Multi-Region and Residency#

Prompt: "We're launching in the EU with a data-residency requirement. How do uploads and storage change?"

8. Incident Walkthroughs#

Deep Dive 1: Peak-Traffic Incident — New Year's Eve Upload Surge#

Context: At midnight in each timezone, uploads jump from 600/s to 9,000/s for about 20 minutes. Tonight, media.pending_age_p99 climbs from 3 seconds to 14 minutes in Asia-Pacific, and users complain their posts "aren't showing up". The on-call escalates to you.

Questions to Surface First:

• Is the delay in uploads (session completion), the queue (event lag) or processing (worker throughput)?
• Are new uploads sharing a lane with backfills or reprocessing jobs?
• Is the processing fleet scaling, and on what signal?
• Is one dependency — metadata writes, blob writes, KMS for key wraps — slower than usual?

Typical L5 Approach: Doubles the processor fleet. The new pods start in 4 minutes and pick up backfill events first, because a format backfill had 30M events queued in the same queue. Pending age keeps rising.

Staff Approach: Reads the pipeline stage by stage: sessions completing normally, queue lag 14 minutes, processors saturated — on a re-encode backfill that started at 22:00. Pauses the backfill, confirms new uploads drain within 6 minutes, then splits lanes so new uploads always get priority.

Principal Approach: Treats midnight in every timezone as a known, scheduled peak: a calendar of predictable surges, pre-scaling 30 minutes ahead, a change freeze on backfills during declared peaks, and a capacity review that sizes processing for the worst midnight of the year, not the average day.

Key Takeaway: "Never let work users are waiting for share a queue with work nobody is waiting for."

What clears the Staff bar:

• Localizes the delay to a pipeline stage before adding capacity
• Finds the competing workload instead of scaling blindly
• Turns lane separation into a permanent rule

Deep Dive 2: Silent Failure — Originals That Were Never Kept#

Context: A user asks to download the full-resolution original of a photo from eight months ago. The download returns the 1440-pixel display version. Investigation finds that for 11 weeks last year, about 3% of uploads stored only derivatives — the canonical original write failed silently.

Questions to Surface First:

• Which code path writes the original, and what happens when it fails?
• Does READY require the original to exist, or only the derivatives?
• How many photos are affected, and do any copies of the originals exist anywhere — staging, clients, backups?
• Why didn't any metric notice?

Typical L5 Approach: Fixes the bug so new uploads store originals. Tells affected users the originals are lost.

Staff Approach: Finds that the original write was best-effort after derivatives, with errors logged but not failing the job. Fixes the READY condition to require the original, its checksum and every eager derivative. Then recovers: staging parts were already aborted, but the backup product still held client-side copies for 40% of affected users; a re-upload prompt reaches the rest whose phones still have the file.

Principal Approach: Establishes that "stored" must be verified, not assumed: a continuous audit samples photos and checks every expected object exists with the right checksum, and the durability SLO is measured by that audit rather than by the storage provider's design number.

Key Takeaway: "Durability starts with the write you didn't verify. Audit existence, not just availability."

What clears the Staff bar:

• Makes READY an invariant over every expected object
• Uses every possible recovery source before declaring loss
• Adds a continuous existence audit

Deep Dive 3: Large-Customer Onboarding — A Photo-Book Partner Uploading 2 Billion Images#

Context: A printing partner will migrate 2 billion customer photos (average 5 MB, 10 PB) into the platform over 60 days through a bulk API, starting next month. Sales has signed. You're asked to make it work without hurting regular users.

Questions to Surface First:

• 10 PB in 60 days is ~2 GB/s sustained — what does that do to processing, storage tiers and cost?
• Do these photos need eager derivatives, or are most never viewed?
• Do we need full sessions, or can the partner push from their own object storage?
• How do deletion and residency obligations transfer with the data?

Typical L5 Approach: Gives the partner an API key and the normal upload endpoint. The partner's 400 parallel workers saturate the processing fleet within an hour and regular users' time-to-READY goes to minutes.

Staff Approach: Treats it as a bulk import, not uploads: the partner stages files in their own bucket; a server-side copy job pulls them in at a scheduled rate into the backfill lane; only thumbnails are rendered eagerly, other sizes on demand; originals go straight to the warm tier because they are historical; per-user keys and residency tags are applied at import.

Principal Approach: Turns bulk ingestion into a product capability with a price — imported bytes, processing and five-year storage costed into the contract — and a standard import path every future partner uses.

Key Takeaway: "Bulk import is not a lot of uploads. Give it its own path, its own lane and its own price."

What clears the Staff bar:

• Separates bulk import from interactive uploads
• Prices five years of storage, not 60 days of ingest
• Tunes derivatives and tiers to how imported photos are actually used

Deep Dive 4: Post-Mortem — Home Locations Exposed Through Photo Metadata#

Context: A journalist reports that downloading public photos from the platform reveals GPS coordinates in EXIF data, including for accounts belonging to people at risk. You're leading the post-mortem.

Questions to Surface First:

• Which outputs contain EXIF — derivatives, originals, or both? Which endpoints serve originals?
• Since when, and how many public photos?
• Was stripping ever implemented, and where did it stop working?
• Are cached copies on the CDN also affected?

Typical L5 Approach: Adds EXIF stripping to the resize step for new uploads.

Staff Approach: Finds stripping worked for derivatives, but a "download original" feature added last year served the raw original, EXIF included, to any viewer of a public photo. Disables original download for non-owners immediately, purges cached originals from the CDN, re-renders and re-strips derivatives produced by an affected processor version, and makes originals owner-only.

Principal Approach: Treats location metadata as personal data under the privacy program: a data inventory entry, a review gate for any feature that serves originals, and an automated scanner that downloads samples from public endpoints and fails a release if any output contains GPS tags.

Key Takeaway: "Stripping metadata in one code path is not a guarantee. Scan what you actually serve."

What clears the Staff bar:

• Finds the path that bypassed stripping instead of re-fixing the one that worked
• Purges cached copies, not just the source
• Adds output scanning as a release gate

Deep Dive 5: Multi-Region Expansion — Moving 3 PB of EU Users' Photos Home#

Context: The company commits to EU data residency. 40 million EU users' photos — 3 PB of originals plus derivatives, metadata and backups — currently live in a US region. Deadline: six months.

Questions to Surface First:

• What counts as the user's data — originals, derivatives, metadata, backups, logs, keys?
• Can uploads go to the EU before migration finishes, and how do reads find each photo?
• How do we prove nothing remains in the US afterwards?
• What happens to shared content — an EU user's photo on a US user's feed?

Typical L5 Approach: Copies the EU users' buckets to an EU region and switches the read path. Forgets backups, the US-resident key service and analytics exports that include photo metadata.

Staff Approach: Sets a home region per user. New uploads for EU users go to the EU from day one. A migration copies originals and metadata user by user, verifies checksums, flips the user's location pointer, and only then deletes US copies. Per-user keys are re-wrapped under EU-resident KEKs; US backups become unreadable once the US copies of those keys are destroyed.

Principal Approach: Defines residency as a user attribute every platform must honor — feed caches, search, analytics, ML training sets — and stands up an automated residency audit that plants marker photos and proves they exist only in the home region.

Key Takeaway: "Residency includes backups and keys. Move the key, and the old copies stop mattering."

What clears the Staff bar:

• Enumerates every data class, including backups and keys
• Migrates per user with verification and pointer flips
• Uses key relocation to neutralize unrewritable backups

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Design resumable, chunked upload sessions that are idempotent on a client-generated ID and clean up after themselves
• Define a visibility contract — PENDING to READY — and explain why fan-out keys off READY
• Treat image processing as an untrusted-input boundary with pre-decode limits, sandboxing, re-encoding and quarantine
• Choose eager vs on-demand derivatives from request data, with coalescing for viral misses
• Explain why dedup scope must match encryption scope, and why global dedup leaks existence
• Model storage cost over five years and design age-based tiering with erasure coding
• Enumerate every copy of a photo and design a deletion pipeline with an SLA, CDN purge and crypto-shredding for backups
• Handle residency by home region for bytes, metadata, processing and keys

The Bar for This Question#

Mid-level (L4): Uploads files through the API to object storage and resizes them in a background job. Works in a demo. No resume, no visibility state, no tiering, delete removes one object.

Senior (L5): Presigned uploads, an event-triggered thumbnail function, a CDN, maybe SHA-256 dedup. The gap: retries duplicate photos, followers see broken images during processing, decoders trust input, storage stays hot forever, global dedup leaks privacy, and delete leaves derivatives, cached copies and backups behind.

Staff+ (L6): Frames the problem as a photo's lifecycle within the first five minutes. Designs idempotent resumable sessions, a READY gate with fan-out on ready, a sandboxed pipeline with pre-decode limits, a hybrid derivative strategy, age-based tiering with real cost numbers, per-user dedup, and a deletion pipeline with an SLA and crypto-shredding. Names who pays — client teams for resume logic, users of very old photos for restore latency, the platform for the deletion guarantee. The interviewer should learn something from the answer.

10. Hot Takes#

10.1 "Upload Complete" Is the Wrong Event to Build On#

The Staff position: Downstream systems subscribe to photo.ready and photo.deleted, never to storage events.

Why this matters in interviews: It's the single sentence that removes an entire class of broken-image and privacy bugs.

10.2 Most Derivatives Are Never Viewed#

The Staff position: The eager set is decided by request logs, not by a design doc's list of devices.

Why this matters in interviews: It shows you think in bytes × objects × years.

10.3 Global Dedup Is a Privacy Bug Wearing a Cost-Saving Costume#

The Staff position: Per-user dedup for originals; global dedup only for content that is already public.

Why this matters in interviews: Interviewers float global dedup to see whether you notice the leak.

10.4 The Storage Bill Is Decided by Policy, Not Technology#

The Staff position: The choices that matter — keep originals, when to tier, what's eager — are product and finance decisions. Make them explicit.

Why this matters in interviews: Candidates who pick storage technologies but not policies miss where the money goes.

10.5 "Delete" Without Crypto-Shredding Is a UI Feature#

The Staff position: If backups can't be rewritten per user, the only honest erasure is destroying the key that encrypts that user's data. Design the key hierarchy for deletion on day one; retrofitting it means re-encrypting petabytes.

Why this matters in interviews: Asking "what about the backups?" is the follow-up that separates a feature from a guarantee.

11. Beyond Staff: The Principal View#

Why L7 Sees This Problem Differently#

The Staff engineer builds a correct, cost-aware photo pipeline. The Principal engineer notices that posts, messaging, stories, marketplace and customer support each ingest media independently — five upload protocols, three processing pipelines, two of which don't strip location data, and no single place that can answer "delete everything this user ever uploaded". The L7 problem is one media platform for the company: one upload SDK, one visibility contract, one processing sandbox, one tiering policy and one deletion guarantee, with product teams owning only what their media means.

The Org-Level Fault Line#

One media platform vs per-product media handling.

🧭 Principal Move: "The platform owns upload sessions, processing, storage tiers, delivery and deletion. Products own derivative specs and moderation gates, declared in config. Direct writes to media buckets from product services are blocked after a two-quarter migration, because a bucket nobody tiers and nobody deletes from is a liability."

Cost Model#

Assumptions: rented object storage at illustrative $20/TB-month hot, $10 warm, $2 cold; CDN egress at illustrative blended $0.01–0.02/GB at volume; fully loaded engineer ~$250K/year.

The pricing insight: cost is dominated by bytes at rest and bytes served, not compute. At growth scale, moving aged originals to warm storage and keeping the CDN hit rate above 95% are worth more than every processing optimization combined. Model the five-year curve: storage cost in year five is roughly five times year one at constant prices, unless tiering bends it.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-MEDIA-001: Media Ingestion, Storage and Deletion Standard
Status: Approved   Owners: Media Platform + Privacy + Security

Scope
  Every system that accepts, stores or serves user-uploaded images.

MUST
  1. Ingest through the Media Platform upload SDK and session service;
     direct writes to media buckets are blocked.
  2. Treat a media ID as referenceable only after photo.ready; consume
     photo.deleted and remove derived copies within the deletion SLA.
  3. Decode only inside the platform sandbox with published limits.
  4. Strip location and device identifiers from every output served to
     anyone other than the owner.
  5. Encrypt originals under per-user data keys from the company KMS;
     dedup scope must not exceed key scope.

SHOULD
  1. Declare derivative specs in config; prefer on-demand for rare sizes.
  2. Tag all media with a home region at creation.
  3. Register backfills in the preemptible lane only.

Exceptions
  Filed with Media Platform; privacy review required for any exception
  touching originals or metadata; time-boxed to two quarters.

Success metrics
  - Time-to-READY p95: < 5s for photos under 10 MB
  - Deletion SLA met: > 99.9% of requests (live systems 30 days, backups 60)
  - Products ingesting media outside the platform: 0 by end of year 2
  - Storage cost per monthly active user: tracked, trending down

What I'd Tell the VP#

"Photos are the fastest-growing cost we have — about 45 petabytes of new originals a year — and five teams handle them five different ways, which means we can't prove a deletion and we patch image-decoder vulnerabilities in five places. I'm proposing one media platform: about six engineers for a year. It roughly halves storage cost growth through tiering, gives us a single, auditable deletion guarantee, and removes most of our exposure to malicious-file attacks. Product teams keep control of how their media looks; the platform owns how it's stored, processed and deleted. The main risk is migration effort, so I'd start with posts, which is 70% of the bytes."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll build a great photo pipeline for posts" — correct, but ignores four other teams ingesting media.
• "Delete runs through a pipeline with a CDN purge" — good, but no SLA or evidence for an auditor.
• "We'll tier originals to save cost" — no five-year model or ownership of the thresholds.

Appendices

Appendix A: Mechanics in Depth#

A.1 The Upload Session#

create(upload_id, owner, size, type, sha256?):
  if s = sessions.get(upload_id):            # idempotent
      assert s.owner == owner
      return s
  if sha256 and p = photos.find_ready(owner, sha256):   # per-user dedup
      return new_photo_ref(owner, p.blob_id, upload_id) # refcount + 1, no bytes
  s = sessions.put_if_absent(upload_id, {owner, size, type,
        part_size = choose_part_size(size, network_hint),   # 4–16 MB
        staging_key = "stg/" + random_id(),                  # server-chosen
        state = OPEN, expires_at = now + 7d})
  return s with sign_parts(s, first=4, ttl=15m)

complete(upload_id, parts[]):
  s = sessions.get(upload_id)
  if s.state == COMPLETE: return photos.by_upload(upload_id)  # idempotent
  storage.complete_multipart(s.staging_key, parts)
  verify(size == s.size, sha256 matches if supplied)
  tx:
    sessions.update(upload_id, state=COMPLETE)
    photos.insert(photo_id, owner, upload_id UNIQUE, state=PENDING)
    outbox.append(media.uploaded{photo_id, staging_key})

A.2 The Processor#

on media.uploaded(photo_id, staging_key):
  p = photos.get(photo_id)
  if p.state != PENDING: return                  # duplicate or deleted
  hdr = sniff_header(staging_key)                # magic bytes, declared dims
  reject_if(hdr.format not in ALLOWED or hdr.pixels > 100e6
            or hdr.frames > 500 or size > 50MB)  → QUARANTINED
  in sandbox(mem=2GB, cpu=10s, net=blob_store_only):
      img = decode(staging_key); img = apply_orientation(img)
      orig_key = put_encrypted(original_bytes, key=user_dek(p.owner))
      for spec in EAGER_SPECS:                    # thumb_256, 640, 1080, 1440, blur
          out = encode(resize(img, spec), strip_metadata=True)
          verify_decodes(out)
          put("d/" + sha256(out) + ext, out)      # content-addressed, idempotent
  photos.cas(photo_id, PENDING → READY, derivatives=..., orig=orig_key)
  emit photo.ready
on failure: retry ≤ 3 with backoff, then FAILED or QUARANTINED

A.3 Deletion Workflow#

delete(photo_id):    state → DELETED, deleted_at = now; emit photo.deleted
                     enqueue purge(surrogate_key = photo_id)       # minutes
after 30 days:       if not restored and not legal_hold:
                       delete derivatives; blobs.decref(orig_blob)
                       if refcount == 0: delete object (all replicas/fragments)
account_erasure(u):  run delete for all photos; after bin window:
                       kms.schedule_destroy(user_kek(u))          # backups unreadable
                     write audit record; close SLA timer

Appendix B: Data Model#

CREATE TABLE photos (
  photo_id        BIGINT PRIMARY KEY,
  owner_id        BIGINT NOT NULL,
  upload_id       UUID NOT NULL,
  state           TEXT NOT NULL,          -- PENDING | READY | FAILED | QUARANTINED | DELETED
  content_sha256  BYTEA NOT NULL,
  blob_id         BIGINT,
  width INT, height INT, taken_at TIMESTAMPTZ,
  visibility      TEXT NOT NULL,          -- public | followers | private
  home_region     TEXT NOT NULL,
  deleted_at      TIMESTAMPTZ,
  legal_hold      BOOLEAN NOT NULL DEFAULT false,
  UNIQUE (owner_id, upload_id)
);  -- sharded by owner_id

CREATE TABLE blobs (
  blob_id      BIGINT PRIMARY KEY,
  owner_scope  BIGINT NOT NULL,           -- per-user dedup: blobs never cross owners
  sha256       BYTEA NOT NULL,
  size_bytes   BIGINT NOT NULL,
  tier         TEXT NOT NULL,             -- hot | warm | cold
  key_id       TEXT NOT NULL,             -- wrapped DEK reference
  refcount     INT NOT NULL,
  UNIQUE (owner_scope, sha256)
);

CREATE TABLE derivatives (
  photo_id  BIGINT NOT NULL,
  spec      TEXT NOT NULL,                -- thumb_256 | w640 | w1080 | w1440 | blur
  format    TEXT NOT NULL,                -- webp | jpeg | avif
  object_key TEXT NOT NULL,               -- d/{sha256}.{ext}
  PRIMARY KEY (photo_id, spec, format)
);

Appendix C: Coordination Mechanisms#

C.1 Visibility and Deletion Across Consumers#

C.2 Quick Comparison#

Appendix D: API Contract & Client Behavior#

• Generate upload_id once per file, persist it with the local upload queue, and reuse it on every retry, including after app restarts.
• Compute SHA-256 while reading the file; send it at create so per-user dedup can skip the upload.
• Upload 2–4 parts in parallel on cellular, more on Wi-Fi; retry each part with exponential backoff and jitter; on 403 (expired URL), request fresh part URLs.
• After any interruption, call GET /uploads/{id} and send only missing parts.
• Show the local file immediately; poll or subscribe for READY; never construct derivative URLs client-side.
• Request derivatives with Accept: image/avif, image/webp, image/jpeg; the server picks the best available.
• Private photos: use signed cookies issued per session; never embed long-lived signed URLs in shares.

Appendix E: Observability#

Core metrics:

• media.pending_age_p99{lane}, media.ready_latency_p95, media.failed_total, media.quarantined_total
• upload.session_completion_rate{network, app_version}, upload.sessions_per_unique_sha256
• processor.cpu_util, processor.oom_kills, queue.oldest_age{lane}
• cdn.hit_rate, cdn.origin_requests, resizer.inflight_per_key
• staging.incomplete_upload_bytes, storage.bytes{tier}, storage.cost_per_mau
• delete.purge_pending_age_p99, delete.sla_breach_count, audit.missing_objects_total

Critical alerts:

Debugging the silent failure: overall success rates hide partial writes. Watch the existence audit (every expected object present with the right checksum), the ratio of READY photos to originals stored, and time-to-READY per lane. A photo that's READY without its original looks perfectly healthy until someone asks for the full-resolution file.

Appendix F: Scale Evolution#

What you don't build on day one: on-demand rendering, cold tier with restore UX, residency, perceptual near-duplicate detection, a custom blob store. Each has a trigger in Section 11.

Appendix G: Multi-Tenancy, Fairness & Cost#

• Per-user upload rate limits: e.g. 500 photos/hour interactive, higher for verified backup clients; protects the pipeline from abuse and runaway sync bugs. See the rate limiter.
• Priority lanes: interactive uploads > moderation re-renders > backfills and imports; backfills are preemptible.
• Bulk importers run on a separate path with their own cost tag and rate.
• Cost attribution: storage bytes by tier, CDN egress and processing CPU attributed per product; storage cost per monthly active user reported monthly.
• Abuse: per-user quotas on stored bytes for free tiers; quarantine counts per user feed trust and safety signals.