Technologies referenced in this case study: Apache Kafka · PostgreSQL · Cassandra · Redis · Kubernetes

Related: CDN & Edge Caching · Object Storage · Handling Large Blobs · Job Scheduler · Managing Long-Running Processes · Metrics & Alerting Platform · Buy or Build · Auth & Identity

Reading Guide#

Organized for interview use first, reference second. The upload mechanics (presigned URLs, multipart, the upload state machine) live in Large File Uploads & Delivery, and the edge cache itself lives in CDN & Edge Caching. This page is about the end-to-end video product and what it costs: which renditions exist, when they are made, where they are served from, and who pays for each choice.

Without a cost-and-tail design:
t=0:        Creator uploads a 12-min 4K video. Pipeline encodes 14 renditions
            (H.264 + VP9 + AV1, 144p–2160p) before marking it playable.
t=+38min:   First playable. Creator has already tweeted the link; viewers saw "processing".
t=+40min:   Link goes viral. 2M viewers in 30 min. Edge caches are cold for every rendition.
t=+42min:   Origin egress jumps from 1.5 Tbps to 6 Tbps. Origin p99 TTFB 4s. Rebuffer ratio 3%.
t=+1 month: The same pipeline encoded 14 renditions for 700K other videos that day,
            90% of which got fewer than 50 views. Storage bill +18% month over month.

With fast-path publish, popularity-driven encoding and tiered delivery:
t=0:        Upload completes. Fast path encodes 360p + 720p H.264 in parallel 4s chunks.
t=+90s:     Playable. Remaining H.264 rungs land by t=+8min.
t=+40min:   View velocity crosses threshold → advanced-codec ladder queued at high priority.
            Mid-tier shield coalesces misses: 1 origin fetch per segment per region.
t=+42min:   Origin egress 1.7 Tbps. Edge hit ratio 97%. Rebuffer ratio 0.4%.
t=+1 month: Long-tail videos hold 5 H.264 rungs; sources moved to archive tier after 30 days.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What the Interviewer Is Scoring#

Video streaming is not a storage question. Anyone can put files in object storage.

It is a cost-and-tail question that tests:

• Whether you see that egress, not compute, dominates the bill — and design to move fewer bytes, from closer, more cheaply
• Whether you treat the long tail differently from the head: most uploads are rarely watched, a few are watched millions of times
• Whether you can turn "fast" into numbers: startup time, rebuffer ratio, publish time, glass-to-glass latency
• Whether you know which decisions are made by people outside engineering — licensing, DRM, regional rights, creator expectations

The key insight: Encode once, store a deliberately chosen set of renditions, and serve them from as close to the viewer as possible. Every rendition is a bet: compute and storage now, against egress saved and quality gained on every future view. Staff candidates price that bet per video based on its popularity; Senior candidates encode everything the same way.

One Question, Three Levels#

Positions to Commit To#

Which Problem Are We Solving?#

Three intents produce three different systems. Name them, then commit.

🎯 Staff Move: "I'll design for user-generated uploads at YouTube-like scale, because that's where ingest, the long tail and egress all collide. A premium catalogue spends far more compute per title and pre-positions everything; live trades efficiency for seconds of latency. I'll point out where each would change my design, and you can redirect me to either."

Where the Design Splits#

How Real Companies Built It#

Why this section belongs here: Naming how real video platforms solved these problems shows you've studied the economics, not just the boxes.

YouTube — Custom Transcoding Silicon#

YouTube has said that on average more than 500 hours of video are uploaded every minute, that VP9 takes about 5× more compute to encode than H.264, and that after starting work in 2015 it built a Video (trans)Coding Unit (VCU) — the "Argos" chip — delivering up to 20–33× better compute efficiency than its previous optimized software system on general-purpose servers; the next generation targets AV1 (YouTube blog, ASPLOS 2021 paper).

The paper's extended abstract adds that the VCU fleet spans tens of thousands of servers and that, for offline two-pass encoding, it delivers 8–20× higher throughput than software encoding for H.264 and VP9 respectively (extended abstract).

Staff insight: Better codecs move cost from egress to compute. When upload volume makes software encoding of an advanced codec unaffordable, the choices are: encode it only for popular videos, or change the cost of compute. Saying both out loud — and knowing that only a handful of companies can do the second — is the signal.

Netflix — Per-Title and Per-Shot Encoding#

In December 2015 Netflix described moving from one fixed bitrate ladder for every title to a ladder chosen per title from an analysis of that title's complexity — simple animation needs far fewer bits than grainy action footage to look the same — and later described its Dynamic Optimizer, which chooses encoding parameters per shot to optimize perceptual quality at each bitrate (per-title encoding, Dynamic Optimizer).

Staff insight: Per-title encoding is affordable for Netflix because the catalogue is small and every title is watched many times. For UGC the same idea applies only to the head of the distribution. The interview move is to say where on the popularity curve the extra encode pays back.

Netflix — Open Connect#

Netflix runs its own CDN, Open Connect. Its appliances are provided to qualifying ISP partners at no charge to embed inside their networks, with the same capabilities as the appliances Netflix operates itself, and their content is updated through nightly fills (Open Connect).

Staff insight: Pre-positioning works because a premium catalogue's demand is predictable a day ahead. UGC demand is not, so a UGC platform still needs pull-through caching with a shield. Owning the edge is a decision about decades of traffic, not about this quarter's design. See Content Delivery Network for the cache mechanics.

Apple — HLS Authoring Specification#

Apple's HLS authoring specification for its devices says target segment durations SHOULD be 6 seconds, video segments MUST start with an IDR frame and key frames SHOULD occur every 2 seconds; for Low-Latency HLS the recommended Part Target Duration is 1 second and SHOULD be at least three times the p95 round-trip time; and HEVC variants should use bitrates about 20% below the H.264 values (HLS Authoring Specification, RFC 8216).

Staff insight: Segment duration is not a free parameter — the platform vendor publishes a default, and the low-latency variant ties part duration to network round-trip time. Citing the default and then explaining when you'd deviate is stronger than inventing a number.

Follow-Ups to Expect#

System Architecture Overview#

Reading the diagram: Two almost independent systems share a metadata store. The media pipeline (left) is a batch system judged on publish time and cost per encoded hour. The delivery path (right) is a read system judged on startup time, rebuffer ratio and egress cost. The playback API is the only synchronous control-plane hop on the play path; segments never touch application servers. The number that decides whether the system survives a big night is cdn.byte_hit_ratio — origin load is proportional to the miss rate, not to the audience.

One-Minute Recap#

Numbers to Bring#

All numbers below are this design's working assumptions unless a source is cited. YouTube's own published upload figure is "more than 500 hours every minute" (YouTube blog); the design uses that as its ingest rate.

Interview Walkthrough

A 45-minute plan for "Design YouTube." The goal is to spend under 10 minutes on the boxes and the rest on the three places the system actually breaks: the encode pipeline under load, the edge under a spike, and the cost curve over time.

Phase 1: Requirements & Framing (2–3 minutes)#

Say this, nearly word for word:

"Before I draw anything — there are three different products hiding in 'design YouTube': user uploads at scale, a premium catalogue like Netflix, and live streaming. They share a CDN and a player, but the pipeline and the economics are different. I'll design for user uploads, and I'll call out what changes for the other two.

Scale assumptions: 500 hours uploaded per minute, about 1 billion watch-hours a day, viewers worldwide, mostly on mobile. Functional scope: upload, process, publish, play with adaptive bitrate, basic view counts. Out of scope unless you want them: recommendations, comments, monetization, search.

The constraints I'll design to: a creator sees their video playable within about 2 minutes; viewers start playback in under 2.5 seconds at p95 with under 0.3% of watch time spent rebuffering; and the cost per watch-hour goes down over time, because egress is the line item that decides whether this business works."

Why this works: It names three intents, commits, attaches numbers to "fast", and announces the cost thesis in the first two minutes. The interviewer now expects a conversation about bytes and tails, not about CRUD.

Phase 2: Core Entities & API (1–2 minutes)#

Video        { video_id, owner_id, state, duration_s, source_uri, created_at,
               visibility, rights{regions[], drm_required}, popularity_tier }
Rendition    { video_id, codec, height, bitrate_kbps, state, segment_prefix,
               storage_tier, bytes, created_at, last_served_at }
EncodeTask   { task_id, video_id, chunk_idx, ladder_rung, priority, attempt,
               lease_owner, lease_expires_at }
PlaySession  { session_id, video_id, user_id?, device, cdn, started_at }   # analytics

POST /videos                         → { video_id, upload_id }            # starts multipart
PUT  <presigned part URL>            → 200                                # client → storage
POST /videos/{id}/complete           → 202 { state: PROCESSING }
GET  /videos/{id}/play?device=...    → { manifest_url (signed, 6h), license_url?,
                                         cdn_hints[], start_rung }
GET  <cdn>/v/{id}/{rendition}/seg_{n}.m4s   → segment (immutable, cache 30d)
POST /qoe/beacons                    → 204   # startup, stalls, bitrate switches

Say: "Segments are immutable and content-addressed by video, rendition and index, so they cache forever. Manifests for VOD are cacheable too once the ladder is final; I'll give them a short TTL while rungs are still being added. The play endpoint is the only dynamic call on the play path — it signs URLs, checks rights and region, and hands the player CDN hints."

Phase 3: High-Level Architecture (≤5 minutes)#

Staff candidates spend under 5 minutes here. Draw two lanes: the media pipeline and the delivery path. Everything else is detail you add on demand.

Narrate: "Upload goes straight to storage with presigned multipart — the API never sees the bytes; that pattern is covered in large-blob handling. Completion emits an event. The scheduler splits the source into GOP-aligned 4-second chunks and fans out tasks by priority: fast-path rungs first, the rest of the H.264 ladder next, advanced codec only when popularity says so. Workers write CMAF segments to the rendition origin. On the read side: player calls the play API, gets a signed manifest, fetches segments through edge and shield caches."

Phase 4: Transition to Depth (1 minute)#

"The boxes are standard. The interesting decisions are three: how much encoding we do and when, because that sets publish time and the compute-plus-storage bill; how we protect origin when a video or a premiere goes viral, because that's where outages come from; and how ABR and segment length trade latency against cost. I'd like to start with encoding economics — is that where you want to go, or would you rather start with delivery?"

Phase 5: Deep Dives (25–30 minutes)#

Deep dive 1 — Encoding economics (8–10 min). Walk the arithmetic. 720K hours/day × ~3 core-hours per hour for the H.264 ladder ≈ 2.2M core-hours/day ≈ 90K cores. Advanced codec at ~5× would add ~450K cores if applied to everything. Then the popularity curve: "If 10% of videos earn about 90% of watch time — a typical UGC skew I'd verify in our own data — applying the advanced codec to that 10% costs ~45K cores and saves ~30% of bytes on ~90% of egress. Applying it to the other 90% costs ~400K cores to save bytes on 10% of egress. So the threshold is a number, not a philosophy." Mention chunked parallel encoding for publish time and the priority classes.

Deep dive 2 — Delivery under a spike (8–10 min). Edge + mid-tier shield; request coalescing (one upstream fetch per object per tier while others wait); pre-warm the first 3 segments of the top rungs for anything the recommendation system is about to push; origin sized for 3× normal miss traffic. "If edge byte hit ratio drops from 97% to 91%, origin load triples. That's the incident, so cdn.byte_hit_ratio per region is a paging metric." Defer cache mechanics to the CDN case.

Deep dive 3 — ABR, segments and QoE (6–8 min). 4s segments for VOD (Apple's spec defaults to 6s; I'll go slightly shorter for faster adaptation on mobile). Throughput-based start at a conservative rung, buffer-based steady state with hysteresis. QoE beacons → startup time, rebuffer ratio, bitrate, sliced by ISP and CDN. Server-side hints choose the CDN; the client chooses the rung.

If time remains — live: real-time ladder at ingest, 1s LL-HLS parts, glass-to-glass 3–5s, and why a 10M-viewer spike is a coalescing problem, not a transcoding problem.

Phase 6: Wrap-Up (2–3 minutes)#

"To summarize: the media pipeline optimizes publish time and cost per encoded hour by spending effort according to popularity. Delivery optimizes origin protection and egress cost with tiered caches and coalescing. QoE is measured on the client and sliced by ISP and CDN. What I'd build later: per-title encoding for the head, a hybrid edge once egress crosses tens of Tbps, and a retention policy for the dormant tail written with legal. The one-way doors are the segment format and the DRM scheme; everything else can change behind the manifest."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

"Design YouTube" is the most common media prompt because it contains every cost tension of a large-scale read system in one product. The ingest side is a batch pipeline with a backlog problem. The read side is a cache with a 1,000:1 popularity skew. The bill is dominated by a resource — egress bandwidth — that most backend engineers never see on a dashboard. A Senior engineer can build every box. The Staff signal is noticing that the boxes are not where the risk is: the risk is a viral upload that the encode queue can't absorb, a premiere night with cold caches, and a storage curve that grows faster than watch time. Interviewers use it to see whether you can put a dollar sign and an owner on each of those.

1.2 The L5 vs L6 Contrast — Visual#

The Senior path is correct and ends at scaling the metadata database — which is a few thousand QPS and never the bottleneck. The Staff path spends its time where the money and the outages are.

1.3 The Staff Question That Cuts Through Everything#

"For a video that will be watched N times, what is the cheapest set of renditions — and the cheapest place to serve them from — that still meets our startup and rebuffer targets?"

Every fault line is a version of this question. Encode up front or on demand: depends on N. Per-title encoding: pays back above some N. Own the edge: depends on the sum of N across the catalogue. Segment duration: trades request cost against latency per view. The candidate who asks it out loud has reframed the interview from "what boxes" to "what is each byte worth."

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Intent 1: User-generated uploads at scale. Ingest is enormous and unpredictable: 500 hours a minute, spiky around events and time zones. Most uploads are watched rarely; a small fraction are watched enormously and unpredictably — a video can go from 0 to 5M views in an hour. Creators measure the platform by time-to-playable and by quality on their own phone. The design centre is a priority-aware encode pipeline, popularity-driven encoding effort and pull-through caching with a shield because you cannot predict which video to pre-position.

Intent 2: Premium catalogue. Tens of thousands of titles, each expensive to license and watched by millions. Releases are scheduled, so demand is predictable to the hour. Spending hours of compute per title on per-shot optimization pays back across millions of views. Licensors impose DRM, regional windows and output-protection rules; the license server is on the startup critical path. The design centre is encode quality per bit and pre-positioning — fill edge caches overnight before the release.

Intent 3: Live streaming. No time to optimize encodes — the ladder is produced in real time at ingest. No time to pre-position — the content didn't exist a second ago. Audiences spike from 10K to 10M in minutes when a match goes to penalties. The design centre is latency budget (glass-to-glass), ingest redundancy (dual encoders, dual ingest points), and request coalescing at every tier, because every viewer requests the same newest part within the same second.

🎯 Staff Move: "These three share a player and a CDN but not a cost model. UGC is a tail problem, premium is a quality-per-bit problem, live is a latency problem. I'll design UGC, and for each fault line I'll say in one sentence what the premium and live answers would be."

2.2 When NOT to Build Your Own Video Pipeline#

Most companies that need video should not build any of this.

🎯 Staff Insight: The build-vs-buy line is set by egress volume and by whether video quality is a competitive differentiator. Until both are true, buying is cheaper and better. See Buy or Build for the general framework.

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. Where the Design Splits#

Each fault line below has options, who pays, a Staff default and when to deviate. The question underneath all five is the one from 1.3: for a video watched N times, what is the cheapest way to meet the QoE targets?

3.1 Fault Line 1: Encode Everything Up Front vs Encode on Demand#

The tension: Encoding every rendition at upload makes every first view perfect and wastes compute and storage on videos nobody watches. Encoding on demand saves both and makes the first viewer of a cold rendition wait — or watch a lower rung.

The arithmetic that sets the threshold. An advanced-codec ladder costs about 5× the H.264 encode: roughly 15 core-hours per video-hour, about $0.30 at $0.02 per core-hour, so ~$0.05 for a 10-minute video. It saves about 30% of bytes. A full view of that video at 2.5 Mbps delivers ~190 MB; 30% saved is ~56 MB, worth ~$0.0003 at a blended $0.005/GB. Compute alone breaks even at ~180 full views. Two corrections push the number up: the average view watches maybe 40% of the video (→ ~450 views), and the extra rungs (~0.45 GB for 10 minutes) cost ~$0.05 a year in warm storage, roughly doubling the cost to recover. That lands near 1,000 views — hence the threshold of 1,000 views in 24 hours, lowered when the platform predicts the video is trending.

Staff default: universal H.264 ladder for every upload, fast path first; advanced codec driven by view velocity; dormant videos shrink to two rungs.

When to deviate:

• Premium catalogue: encode everything exhaustively up front — every title is in the head.
• Archive or compliance video: store the source only; encode on first request and accept a 30–60s wait.
• Encode hardware makes advanced codecs cheap: if per-hour encode cost drops 10–20×, the threshold drops by the same factor and "encode everything" becomes reasonable again.

🎯 Staff Move: "I won't pick 'encode everything' or 'encode on demand' on principle. Advanced-codec encoding costs about 30 cents per video-hour and saves about 0.17 cents per watch-hour; with partial viewing and a year of storage for the extra rungs, it pays back at roughly 1,000 views. That's my threshold, and I'd re-derive it whenever compute or egress pricing changes."

3.2 Fault Line 2: Fixed Ladder vs Per-Title or Per-Shot Encoding#

The tension: A fixed ladder (same bitrates for every title) is predictable and cheap to compute. It overspends bits on simple content — slides, animation, a talking head — and underspends on grainy, high-motion content. Per-title and per-shot encoding fix both by running analysis or trial encodes, which costs compute per title.

Who pays: the encode budget pays up front; the delivery budget and the viewer's data plan collect the savings on every view. Live can't use either — there's no time for trial encodes — so live ladders are fixed and slightly over-provisioned.

Measuring quality: you need a perceptual metric (VMAF, SSIM or an internal model) to say "equal quality". Without one, per-title encoding is guesswork and every ladder change is an argument. Treat the metric pipeline as a prerequisite, owned by the media pipeline team.

Staff default: complexity-classified ladders for every upload (cheap, catches screen recordings and animation), full per-title encoding for the head, per-shot never for UGC unless hardware changes the cost.

🎯 Staff Insight: Per-title encoding is a bandwidth project funded by the encode budget. Present it with both numbers — extra core-hours and saved petabytes — or it will lose every prioritization meeting to features.

3.3 Fault Line 3: Own CDN vs Commercial CDN vs Both#

The tension: A commercial CDN is instant, global and someone else's pager. At very large egress volumes, owning cache servers — in your own points of presence and, further, embedded inside ISP networks — lowers unit cost and improves QoE, but it is a multi-year capital program with hardware, logistics and peering negotiations.

Rough break-even. At ~$0.01/GB commercial, 1 Tbps sustained average is 10.8 PB/day ≈ $108K/day ≈ $40M/year. An owned edge that serves the same traffic at $0.002–0.004/GB all-in (hardware amortized, colocation, transit/peering, staff) saves roughly $25M/year per Tbps — once the fixed cost of a 40–80 person edge organization ($15–25M/year) is covered. Below a few Tbps average, buy. Above tens of Tbps, owning wins if you can execute. The details of cache design are in Content Delivery Network; the decision framework is in Buy or Build.

Staff default for our design (190 Tbps peak): hybrid. Owned edge and ISP peering carry the steady head; one or two commercial CDNs carry overflow, new markets and the long tail where our caches would thrash.

When to deviate: a startup or mid-size platform should use one commercial CDN with committed pricing and add a second only when spend or a provider outage justifies it.

3.4 Fault Line 4: Segment Duration — Latency vs Efficiency and Request Rate#

The tension: Shorter segments let live streams run closer to real time and let ABR react faster. Longer segments mean fewer HTTP requests, fewer keyframes (better compression), smaller manifests and better cache efficiency.

The request rate matters because CDNs charge per request at some tiers, because each request has a fixed overhead in TLS and HTTP processing at the edge, and because manifests for live are re-fetched every part.

Who pays: short segments are paid for by the edge (request rate) and by bandwidth (keyframe overhead); long segments are paid for by viewers on bad networks (slower adaptation → more stalls) and by live audiences (latency).

Staff default: 4 s segments with 2 s keyframes for VOD (two GOPs per segment gives the packager flexibility); 6 s segments with 1 s parts for low-latency live where latency is in the product promise; standard 6 s segments for live where 20–30 s is acceptable — most live content does not need LL.

🎯 Staff Move: "Segment length is a cost knob, not a latency knob. Halving it doubles request rate across 75 million viewers. I'll use 4-second segments for VOD and reserve 1-second parts for the live events where latency is part of the product, like sports betting or interactive streams."

3.5 Fault Line 5: Client-Driven ABR vs Server-Guided Steering#

The tension: The player knows its buffer level and recent throughput; the platform knows which CDN is healthy, which ISP is congested, what each byte costs and what a million other players are seeing. Pure client ABR is scalable and robust but myopic. Pure server control is informed but adds a control plane to the play path and cannot see the device's buffer.

Staff default: hybrid client ABR with hysteresis (up-switch needs headroom and a hold time; down-switch is immediate), plus server hints delivered with the play response and refreshed every few minutes: preferred CDN order, a starting rung, and an optional bitrate cap per ISP or region during congestion. If the hint service is down, the player uses cached hints or none — never blocks playback. This is a degraded-mode design: the server can improve playback but cannot stop it.

🎯 Staff Insight: The server-side bitrate cap is the most powerful incident lever you have. During an ISP congestion event, capping that ISP's viewers at 720p cuts their egress by ~45% and usually removes the congestion that caused the stalls. Decide in advance who may pull it — the delivery on-call — and who must be told — partnerships, because the ISP will ask.

4. When It Breaks#

4.1 Transcoding Backlog After a Viral Upload Spike#

A global event (an election result, a celebrity incident, a natural disaster) produces a 3× upload surge in 40 minutes, concentrated in long phone videos.

t=0:       Upload rate 500 → 1,400 h/min. Encode demand ~2.8× fleet capacity.
t=+5min:   encode.queue_age_p99 1min → 9min. Fast-path and ladder tasks share one queue.
t=+12min:  Fast-path p50 publish time 90s → 14min. Creators retry uploads (duplicates +20%).
t=+15min:  Page: encode.fastpath_queue_age_p99 > 5min for 5m.
t=+18min:  On-call: pause advanced-codec and re-encode classes (frees ~30% of fleet).
t=+20min:  Scheduler moves full-ladder tasks behind fast path. Autoscaler requests
           preemptible capacity: +40% in 25 min.
t=+45min:  Fast path back under 2 min. Ladder backlog 6 hours, drains overnight.
t=+1 day:  Post-mortem: duplicate uploads (same content hash) were encoded twice.

Detection: encode.queue_age_p99{class}, publish.time_to_playable_p50, encode.tasks_pending{class}, upload.duplicate_content_ratio.

Mitigation: strict priority classes — fast path, ladder, advanced codec, re-encode/backfill — with the lower classes preemptible; dedup by content hash before encoding; shed the advanced-codec class first because it is pure optimization.

Prevention: capacity model with 2× headroom for the fast path only (cheap: it's 2 rungs); preemptible spot capacity for everything else; quarterly load test replaying a 3× surge. Scheduling mechanics are in Job Scheduler and Managing Long-Running Processes.

Owner: media pipeline on-call; capacity planning owns the headroom model.

4.2 Cold Cache on a Premiere Night#

A scheduled episode releases at 20:00 local. 8M viewers press play in the first five minutes.

t=-24h:    Nobody pre-warmed: the release workflow publishes metadata at 20:00 only.
t=0:       8M play requests in 300s. Every edge cache is cold for every rung.
t=+5s:     Edge miss rate 100% for segment 1. Mid-tier coalesces per region, but 600 edge
           sites × 6 rungs × first 5 segments each go upstream.
t=+20s:    Origin egress 0.4 → 5 Tbps. Origin TTFB p99 300ms → 4s.
t=+30s:    Startup p95 2s → 9s. Players time out and retry → miss traffic doubles.
t=+2min:   Page: qoe.startup_time_p95 > 5s and origin.egress_gbps > 3x baseline.
t=+4min:   Edge caches warm. Hit ratio 97%. Startup recovers. 11% of viewers had given up.

Detection: qoe.startup_time_p95{title}, cdn.byte_hit_ratio{region}, origin.egress_gbps, origin.ttfb_p99, player.start_failure_rate.

Mitigation: request coalescing at edge and shield; origin rate limiting with Retry-After honored by players (with jitter); start viewers on a lower rung for the first 10 seconds during a known spike.

Prevention: release workflow publishes segments to origin hours early and pre-warms the first 60 seconds of the top 4 rungs into every edge site; metadata flips visibility at 20:00. For premium releases, pre-position the whole title overnight.

Owner: delivery team for coalescing and pre-warm; content operations owns the release checklist.

4.3 ABR Oscillation Storm#

A player release changes the throughput estimator to a shorter window. On a congested ISP, a million players see the same dip at the same time.

t=0:       Player v8.2 ships to 30% of Android. Estimator window 10s → 3s.
t=+2h:     Evening peak on ISP-X. Link utilization 92%.
t=+2h01:   Players on ISP-X see throughput dip, switch 1080p → 480p together.
t=+2h02:   Link frees up. Players estimate high throughput, switch back to 1080p together.
t=+2h03:   Link saturates again. Repeat every ~40s.
t=+2h20:   qoe.bitrate_switches_per_min{isp=X} 0.3 → 4.1. Rebuffer ratio 0.2% → 1.8%.
t=+2h25:   Page on rebuffer ratio by ISP. Delivery on-call caps ISP-X at 720p via hints.
t=+2h30:   Oscillation stops. Rebuffer 0.3%. Player rollback started.

Detection: qoe.bitrate_switches_per_min{isp,player_version}, qoe.rebuffer_ratio{isp}, cdn.egress_gbps{isp} showing a sawtooth.

Mitigation: server-side rung cap for the affected ISP; halt the player rollout.

Prevention: up-switch hysteresis (headroom + 20s hold) and randomized hold times so players desynchronize; player releases staged by ISP and device with QoE guardrails per stage — a player is a fleet of 75M distributed control loops.

Owner: player team for the algorithm and rollout; delivery on-call for the cap.

4.4 CDN Region Failure Causes an Origin Stampede#

A commercial CDN loses a large region; traffic shifts to our owned edge and a second CDN, both cold for that region's long tail.

t=0:       CDN A loses region EU-West. DNS/steering moves 3 Tbps to CDN B + owned edge.
t=+30s:    New caches miss on the tail. Origin egress 0.9 → 9 Tbps (10×). Capacity: 3×.
t=+1min:   Origin TTFB p99 8s. Startup failures 6%. Players retry → load climbs.
t=+3min:   Origin protection: priority shedding — lowest 3 rungs always served; 1080p+
           requests get 429 Retry-After 10s with jitter. Steering caps region at 720p.
t=+15min:  Caches warm. Origin 3.5 Tbps. Caps lifted in 25% steps over 20 min.

Detection: cdn.availability{provider,region}, origin.egress_gbps, origin.shed_requests_total, player.start_failure_rate{region}.

Mitigation: priority shedding at origin by rung; steering-based rung cap; stagger traffic moves (25% per minute) rather than flipping a region at once.

Prevention: keep the secondary CDN partially warm by sending it a steady 10–20% share; size origin for the largest single-region failover, not for average miss rate; game day twice a year.

Owner: delivery team; vendor management for the CDN incident report.

4.5 Storage Costs Growing Faster Than Views#

The slow failure: nothing pages, but the storage line grows 40% a year while watch time grows 15%.

What's happening: every upload adds ~3 PB/day of renditions and ~2.6 PB/day of source. Without tiering, year 3 holds ~6 EB, and the bytes are overwhelmingly in videos that earn almost no views. Watch time is concentrated in recent and popular videos; storage is concentrated in old and unpopular ones.

Detection: storage.bytes{tier}, storage.bytes_per_watch_hour (the ratio that should be flat or falling), storage.cost_growth_vs_watchtime_growth reviewed monthly.

Mitigation: lifecycle policy by age and views: source to archive at 30 days (it's only needed for re-encodes); rendition sets shrink as videos cool; advanced-codec rungs deleted after 180 days unserved; dormant videos keep two rungs. Re-encoding from archive on a spike costs minutes of compute — cheap next to years of storage.

Prevention: storage cost per watch-hour as a quarterly KPI with a named owner; retention policy agreed with legal and creator relations. Storage mechanics are in Object Storage.

Owner: media platform for the policy engine; finance partner for the KPI; legal for the retention promise.

4.6 Operational Reality Matrix#

🎯 Staff Insight: Two classes of failures dominate: the ones you see in minutes (cold cache, CDN loss, ABR storms), which are all versions of "origin load is proportional to the miss rate"; and the one you see in quarters (storage drift), which is a policy failure. The first class needs coalescing and shedding; the second needs an owner and a KPI.

5. Scorecard#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Codec trivia ≠ system design. Explaining B-frames, CABAC and motion vectors impresses briefly; if it doesn't lead to a cost or publish-time decision, it's trivia.
• Naming HLS and DASH ≠ understanding delivery. The signal is what segment length does to request rate and latency.
• "Netflix built Open Connect" ≠ a build decision. Without traffic volume and break-even reasoning, citing it suggests copying a company 1,000× your size.
• Kubernetes for transcoding ≠ scheduling design. Running workers on Kubernetes is fine; the signal is priority classes, preemption and chunk-level retry.

6. The 45 Minutes, Phase by Phase#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Upload resumability details — one sentence; link to Handling Large Blobs.
• Codec internals — name codecs and their compute/byte tradeoff only.
• Recommendations, comments, search — scoped out; mention that view counts feed popularity.
• View-count accuracy — a separate aggregation problem; see Ad Click Aggregation for the pattern.
• Metadata sharding — the metadata DB is small; a sharded PostgreSQL or Cassandra is fine, and saying why you're skipping it is the signal.

6.4 Follow-Up Questions to Expect#

1. "How long does a 1-hour 4K upload take to become playable, and how would you make it faster?"
2. "How much storage do you add per day, and what's the bill in year three?"
3. "What does origin see when a premiere starts at 20:00 for 8 million viewers?"
4. "Why 4-second segments? What changes at 2 seconds? At 10?"
5. "How does the player decide which rendition to fetch, and how do you stop it oscillating?"
6. "At what scale would you build your own CDN?"
7. "How do you get live latency under 5 seconds, and what does it cost?"

7. Practice Rounds#

Drill 1: The Opening#

Prompt: "Design YouTube."

Drill 2: The Core Mechanic — Publish Time#

Prompt: "A creator uploads a 1-hour 4K video. How long until viewers can watch it, and how do you make that fast?"

Drill 3: Make It Concrete — Storage per Day#

Prompt: "How much storage do you add per day? What about year three?"

Drill 4: Dependency Down — The Origin Is Struggling#

Prompt: "Origin TTFB p99 just went from 200ms to 6 seconds during peak. What do you do?"

Drill 5: Hot Key — One Video, 5M Viewers in an Hour#

Prompt: "A video uploaded 20 minutes ago goes viral: 5 million viewers in an hour. What breaks?"

Drill 6: Multi-Tenant — Live and VOD on One Edge#

Prompt: "We're adding live sports on the same CDN and edge fleet as VOD. What do you worry about?"

Drill 7: Build vs Buy — Our Own CDN#

Prompt: "Finance says the CDN bill is our second-largest expense. Should we build our own?"

Drill 8: Policy Change Without an Outage — New Codec Rollout#

Prompt: "We want to move the head of the catalogue from VP9 to AV1. How do you roll it out?"

Drill 9: Cost — Cut Delivery 30%#

Prompt: "The CFO wants delivery cost down 30% next year without hurting QoE. Where do you look?"

Drill 10: Multi-Region — Rights and Residency#

Prompt: "We're launching in a country that requires certain content to be blocked and some user data to stay in-country."

8. Incident Walkthroughs#

Deep Dive 1: Peak-Traffic Incident — The Championship Final#

Context: A live final peaks at 11M concurrent viewers, 4× the previous record. At the 80th minute, startup failures hit 9% and existing viewers report stalls. The on-call escalates to you.

Questions to Surface First:

• Is it ingest (the stream itself), packaging, origin, or edge? live.ingest_gap_seconds vs origin.ttfb_p99 vs cdn.byte_hit_ratio.
• Are failures concentrated on one CDN, ISP or device class?
• Are players requesting parts that don't exist yet (clock skew, too-aggressive LL settings)?
• Did anything change: player version, packager config, steering weights?

Typical L5 Approach: Scales origin servers horizontally and asks the CDN to add capacity. Origin recovers partially; stalls continue because the edge is the bottleneck.

Staff Approach: Finds that one CDN's edge is returning 404s for parts requested ~1s before they exist, and the players retry immediately — a retry storm into origin. Shifts 30% of traffic to the second CDN in steps, caps the top rung, and turns on blocking playlist reload so requests wait at the edge instead of retrying.

Principal Approach: Treats it as a readiness failure: the event was 4× the record with no load test at that scale. Establishes an event-readiness review for anything forecast above 2× the previous peak — capacity reservations with each CDN, a rehearsal with synthetic load, and a pre-agreed degradation ladder signed off by the sports partnership owner.

Key Takeaway: "In live, a request for something that doesn't exist yet is the most expensive request you'll serve. Hold it at the edge; never let it become a retry."

What clears the Staff bar:

• Slices by CDN and player version before scaling anything
• Identifies the retry storm as the amplifier
• Uses a pre-approved degradation ladder

Deep Dive 2: Silent Failure — The Ladder That Stopped at 720p#

Context: A creator support ticket says "my videos are blurry." Investigation shows that for 9 days, about 18% of uploads never got 1080p. No alert fired: every video was PLAYABLE.

Questions to Surface First:

• Which uploads: by source codec, resolution, device, region?
• Did 1080p tasks fail, never get created, or get created and starve?
• Why didn't an alert fire — what does "success" mean in our metrics?

Typical L5 Approach: Finds a worker crash on a new phone's HEVC source variant, fixes the decoder flag, re-runs the failed tasks.

Staff Approach: Fixes the crash, then fixes the definition of success: the pipeline measured "video reached PLAYABLE" but nothing measured "video reached its expected ladder." Adds encode.ladder_completeness — the share of videos with every expected rung within 1 hour — as a paging SLO, and a reconciler that compares expected vs actual rungs hourly.

Principal Approach: Generalizes to every derived asset: thumbnails, captions, audio tracks. Establishes "expected vs actual derived assets" as a platform-level reconciliation with a dashboard per asset type, and adds a canary corpus of source files from new devices to the pipeline's release checks.

Key Takeaway: "PLAYABLE is not done. Measure the ladder you promised, not the first rung you shipped."

What clears the Staff bar:

• Distinguishes partial success from success in the metrics
• Finds the ownerless DLQ behind the bug
• Adds reconciliation of expected vs actual outputs

Deep Dive 3: Large-Customer Onboarding — A Broadcaster Brings 400,000 Hours of Archive#

Context: A broadcaster partnership will upload 400K hours of archive in 6 weeks, plus 50 live channels. Their contract requires DRM on all content and geo-restriction to three countries.

Questions to Surface First:

• How does 400K hours compare to daily capacity? (~55% of a normal day's upload volume — but spread over 6 weeks, ~1.3% extra per day.)
• What DRM systems and security levels does the contract require? Who holds keys?
• Which content needs advanced codecs on day one vs on popularity?

Typical L5 Approach: Lets the archive flow through the normal upload pipeline and adds DRM encryption to the packaging step.

Staff Approach: Routes the archive through a bulk-ingest class below normal uploads, rate-limited to keep the fast-path SLO intact. Adds multi-DRM packaging (CMAF cbcs so one encrypted set serves Widevine, FairPlay and PlayReady), a license service with per-title policy (regions, security level), and region checks in the play API. Live channels get dedicated ingest pairs.

Principal Approach: Sees that DRM and rights are now platform capabilities, not a partner integration. Funds a rights-and-licensing service owned jointly by the playback platform and business affairs, so the next partner is configuration, and negotiates the contract's security-level requirements with partnerships before engineering commits.

Key Takeaway: "Bulk work gets its own class with its own quota. A partner's archive must never be the reason a creator waits."

What clears the Staff bar:

• Sizes the archive against daily capacity before designing
• Isolates bulk ingest from the creator SLO
• Encrypts once for all DRM systems

Deep Dive 4: Post-Mortem — The Month Storage Grew 22%#

Context: The monthly cloud bill shows storage up 22% while watch time grew 1%. Finance asks for an explanation and a plan by Friday. You own the post-mortem.

Questions to Surface First:

• Which tier and which object classes grew: source, renditions, multipart leftovers, thumbnails?
• Did a lifecycle rule stop running or change?
• Did upload behaviour change (more 4K, longer videos)?

Typical L5 Approach: Finds that a lifecycle rule was disabled during a migration, re-enables it, and reports the fix.

Staff Approach: Re-enables the rule, then finds three causes: the disabled rule (60% of growth), a new 4K-default camera app doubling average source bitrate (25%), and incomplete multipart uploads never aborted (15%). Adds an abort rule for incomplete multipart uploads older than 7 days, a lifecycle-rule drift check, and storage.bytes_per_watch_hour as a weekly alert.

Principal Approach: Makes storage efficiency a standing KPI owned by the media platform with a finance partner, adds lifecycle rules to infrastructure-as-code with policy checks so they can't be disabled silently, and opens the retention policy discussion with legal — the 4K-source trend will continue regardless of hygiene.

Key Takeaway: "Storage doesn't page. Give it a ratio metric and an owner, or it grows until finance finds it."

What clears the Staff bar:

• Decomposes growth into causes with percentages
• Catches the multipart-leftover class (see Large File Uploads & Delivery)
• Turns a one-time fix into a guardrail

Deep Dive 5: Multi-Region Expansion — Launching in a Market With Weak Transit#

Context: The platform launches in a large market where international transit is expensive and congested at evening peak. Early QoE: startup p95 6s, rebuffer ratio 2.4%.

Questions to Surface First:

• Where is the bottleneck: CDN presence in-country, ISP interconnects, or last-mile?
• What device mix and data-plan constraints apply? (Many viewers on prepaid mobile data.)
• Which content is watched — is the head local or global?

Typical L5 Approach: Adds a cloud region in-country as an origin and points the CDN at it.

Staff Approach: Measures per ISP: the bottleneck is international transit into the two largest ISPs. Moves the head in-country — a commercial CDN with local presence plus, for the two largest ISPs, a direct peering arrangement. Adjusts the ladder for the market: an extra low rung (~150 kbps) and data-saver defaults on mobile. Results: startup p95 2.4s, rebuffer 0.5%.

Principal Approach: Treats market entry as a delivery-strategy decision: when a market's traffic exceeds a threshold, start the owned-edge or ISP-embedded program there; until then, commercial CDN with local presence. Writes the market-entry playbook so QoE targets, ladder adaptations and partnership steps are standard.

Key Takeaway: "In a new market, the bottleneck is usually an interconnect, not a server. Measure per ISP before building anything."

What clears the Staff bar:

• Diagnoses per ISP and hour
• Adapts the ladder to the market's devices and data costs
• Brings in the partnerships owner

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Explain why video streaming is a cost-and-tail problem and put an egress number on it within the first two minutes
• Derive storage per day and encode compute from upload rate, ladder bitrates and core-hours per video-hour
• Design a priority-aware encode pipeline: fast path, ladder fill, popularity-driven advanced codec, bulk and backfill classes
• Derive a popularity threshold for extra encoding from compute cost, byte savings and egress price
• Choose segment length with request-rate and latency numbers, and say when low-latency parts are worth it
• Describe hybrid ABR with hysteresis and the server-side hints that can cap quality during incidents
• Protect origin with shields, coalescing, rung-based shedding and failover sizing
• Price own vs commercial vs hybrid edge and name the traffic level where each wins
• Write a lifecycle and retention policy for the long tail and name who signs off

The Bar for This Question#

Mid-level (L4): Draws upload → storage → transcoder → CDN → player and explains HLS at a high level. Uses a fixed ladder and a single CDN. No numbers beyond "millions of users". Would build something that works for a few thousand videos and has no plan for spikes, cost or the tail.

Senior (L5): Adds chunked upload, a transcoding queue, multiple resolutions, a CDN, adaptive bitrate and a sharded metadata store. Knows HLS and DASH. The gap: treats every video the same, never prices egress, says "short segments" without a request-rate number, and has no origin-protection story for a premiere. Plausible and competent — and would produce a storage bill that grows faster than the business and a 20:00 outage on the first big release.

Staff+ (L6): Frames the problem around egress and the long tail in the first minutes. Splits publish time from full quality. Derives the popularity threshold for advanced encoding. Protects origin with coalescing and rung-based shedding and sizes it for the largest regional failover. Sets QoE SLOs sliced by ISP and device, and names owners for the pipeline, player, delivery and content-ops handoffs. Knows when not to build any of it. The interviewer should learn something from the answer.

10. Hot Takes#

10.1 "Video Is a Networking Business That Happens to Use Storage"#

The Staff position: Optimize for bytes delivered and where they're delivered from. A 10% bitrate saving is worth more than halving the encode fleet.

Why this matters in interviews: Candidates who spend their time on storage layout or the metadata DB are optimizing the smallest line items.

10.2 "Most Videos Should Never Get the Good Codec"#

The Staff position: Encoding effort is an investment that pays back per view. Spend it where views are.

Why this matters in interviews: "Encode everything in AV1" sounds modern; it's a compute bill for videos nobody watches.

10.3 "Low Latency Is a Product Feature You Pay For, Not a Default"#

Standard HLS at 20–30 seconds is fine for most live content. LL-HLS at 3–5 seconds roughly triples to quadruples the request rate per viewer and tightens every origin and edge SLO. WebRTC under 1 second gives up HTTP caching altogether.

The Staff position: Default to standard latency; enable low latency per event where interaction, betting or spoilers make seconds matter — and have product say so.

Why this matters in interviews: "We'll use low-latency everywhere" signals that you haven't priced it.

10.4 "Owning a CDN Is a Ten-Year Decision Disguised as a Cost Project"#

The savings are real at extreme scale, but the commitment includes hardware generations, ISP relationships in hundreds of networks, a logistics operation, and an organization that will exist for a decade. The cheaper levers — codecs, hit ratio, contracts — should be exhausted first.

The Staff position: Build the edge only when traffic is large, predictable and growing, and keep a commercial CDN as both overflow and leverage.

Why this matters in interviews: Citing Open Connect without the volume math is copying; citing it with the math is judgment.

10.5 "Rebuffer Ratio Is the Only Metric the CEO Needs"#

Startup time, bitrate, resolution and start failures all matter to engineers. Stalls are what viewers remember and what most directly ends sessions.

The Staff position: One headline QoE metric — rebuffer ratio, sliced by ISP and device — with the rest as diagnostics. Every cost-cutting change ships with a rebuffer-ratio guardrail.

Why this matters in interviews: Naming one metric that leadership can track shows you can communicate a technical system upward.

11. Beyond Staff: The Principal View#

Why L7 Sees This Problem Differently#

The Staff engineer designs a pipeline and an edge that meet QoE targets at a defensible cost. The Principal engineer notices that the company's margin is set by cost per watch-hour, that it is driven by three levers with very different time horizons — codec efficiency (quarters), edge ownership (years) and retention policy (a legal and creator-relations decision) — and that the company currently runs three video stacks: VOD, shorts and live, each with its own encoder settings, its own CDN contract and its own idea of quality. The L7 problem is setting the video platform's economic and organizational shape for the next three to five years: one pipeline with workload classes, one edge with per-product quotas, one QoE definition, and a cost-per-watch-hour target that every team can see.

🧭 Principal Move: "I'd make cost per watch-hour and rebuffer ratio the two numbers every video team reports, and fund the levers in order of payback: advanced codec for the head this year, a hybrid edge in our top markets over three years, and a retention policy for the dormant tail that legal and creator relations sign. Each lever has a different owner, so the plan has to be written as a portfolio, not as one project."

The Org-Level Fault Line#

One video platform vs per-product video stacks.

🧭 Principal Insight: The platform should own everything whose cost scales with bytes — encode infrastructure, packaging, edge, contracts — and expose knobs for everything whose value differs by product: ladder templates, latency class, startup strategy.

Cost Model#

Assumptions: average delivered bitrate 2.5 Mbps; source 8 Mbps; H.264 ladder 9.2 Mbps total; software encode ~3 core-hours per video-hour at $0.02 per core-hour; storage blended across tiers; fully loaded engineer ~$250K/year.

The pricing insight: at every scale in this model, egress costs roughly 50× or more what encode compute costs. A team of 5 engineers (~$1.25M/year) that lowers average bitrate 5% saves ~$0.5–1.2M/month at growth scale and ~$3.5–7.5M/month at hyperscale. At startup scale the same team saves ~$10K/month — buy the managed service and spend the engineers on product.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-VID-001: Video Encoding, Packaging and Delivery Standard
Status: Approved   Owners: Video Platform + Delivery Engineering

Scope
  Every product that stores or serves video: VOD, shorts, live, ads, previews.

MUST
  1. Encode through the shared pipeline using a workload class
     (fastpath, ladder, advanced, bulk, backfill); no product runs its own encoders.
  2. Package as CMAF; encrypt with cbcs where DRM is required.
  3. Serve segments only through the platform edge or contracted CDNs via steering;
     no direct origin URLs to clients.
  4. Emit standard QoE beacons (startup, stalls, bitrate switches, start failures)
     with player version, device, ISP and CDN dimensions.
  5. Keep the source for every video; treat renditions as regenerable.
  6. Attach a lifecycle policy to every rendition set at creation.

SHOULD
  1. Use 4–6 s VOD segments with 2 s keyframes; 1 s parts only for low-latency live classes.
  2. Gate advanced codecs on device hardware decode capability.
  3. Ship player changes in stages per device family with rebuffer-ratio guardrails.

Exceptions
  Filed with Video Platform; decided within 10 business days; time-boxed to 2 quarters.
  Exceptions to MUST 5 need legal sign-off.

Success metrics
  - Cost per watch-hour: −15% year over year
  - Rebuffer ratio ≤ 0.3% globally, ≤ 0.6% in every market with > 1M daily viewers
  - Publish time p50 ≤ 2 min for uploads under 15 min
  - Storage bytes per watch-hour: flat or falling quarter over quarter
  - Products running their own encoders or CDN contracts: 0 by end of year 2

What I'd Tell the VP#

"Delivering video is our largest cost after content, and it scales with every hour watched. We can bend that curve with three levers on three timelines: better compression for our most-watched videos this year, a lower-cost delivery network in our biggest markets over three years, and a clear retention policy for videos nobody watches. Together they target a 15% annual reduction in cost per hour watched while holding stall rates under 0.3%. The first lever needs about five engineers; the second is a multi-year capital program I'd bring back as a separate proposal with market-by-market numbers. The retention policy needs legal and creator relations, not engineering."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll use AV1 to save bandwidth" — correct lever, no payback model, no device-capability plan, no owner.
• "We'll build our own CDN like Netflix" — no traffic threshold, no fixed-cost estimate, no plan for markets without boxes.
• "We'll keep everything in cold storage" — ignores that the retention promise is a policy decision with creators and legal.

Appendices

Appendix A: Mechanics in Depth#

A.1 The Chunked Transcode DAG#

def plan(video):
    if dedup.exists(video.content_hash):                 # re-upload of identical bytes
        return link_renditions(video, dedup.get(video.content_hash))
    chunks = split_points(video.source, target_s=4)      # closed GOP boundaries only
    for rung in FASTPATH:                                 # 360p, 720p H.264
        for i in range(len(chunks)):
            enqueue(task(video.id, rung, i), cls="fastpath")
    for rung in LADDER - FASTPATH:
        for i in range(len(chunks)):
            enqueue(task(video.id, rung, i), cls="ladder")

def run(task):                                            # idempotent per (video, rung, chunk)
    key = f"{task.video_id}/{task.rung}/{task.chunk}"
    if output_exists(key):                                # replay after preemption → no-op
        return
    with lease(key, ttl="2m", heartbeat="20s"):
        out = encode(read_chunk(task), task.rung)
        put_if_absent(key, out)                           # write-once segment objects
    if all_chunks_done(task.video_id, task.rung):
        package_and_publish(task.video_id, task.rung)     # single writer per rung

The idempotent-task and lease mechanics are general; see Workflows, Sagas & Compensation and Job Scheduler.

A.2 Hybrid ABR With Hysteresis#

def next_rung(state, rungs):
    tput = ewma_throughput(window_s=10)                   # bytes/s, both fast and slow EWMA, take min
    if state.phase == "startup":
        return highest(rungs, bitrate <= 0.7 * tput)      # conservative start
    if state.buffer_s < 3:
        return rungs.lowest                               # panic
    if state.buffer_s < 8:
        return highest(rungs, bitrate <= 0.8 * tput)      # immediate down-switch
    up = rungs.above(state.current)
    if (up and state.buffer_s > 25 and tput > 1.4 * up.bitrate
            and now() - state.last_switch > hold_s()):     # hold 20s + random 0–10s
        return up
    return min(state.current, hints.rung_cap or rungs.highest)

The random component in the hold time desynchronizes players that share a bottleneck — the fix for the oscillation storm in 4.3.

A.3 Live Pipeline#

Budget for 4 s glass-to-glass with LL-HLS: capture + contribution ~500 ms, real-time encode ~500 ms, packaging one 1 s part ~1 s, CDN ~200 ms, player buffer ~2 parts ≈ 2 s.

Appendix B: Data Model#

videos           (video_id PK, owner_id, state, duration_s, source_uri, source_tier,
                  content_hash, visibility, rights_regions[], drm_required,
                  popularity_tier, created_at, last_viewed_at)
renditions       (video_id, codec, height, bitrate_kbps, state, segment_prefix,
                  storage_tier, bytes, created_at, last_served_at,
                  PRIMARY KEY (video_id, codec, height))
encode_tasks     (task_id PK, video_id, rung, chunk_idx, class, attempt,
                  lease_owner, lease_expires_at, status)
view_counters    (video_id, hour_bucket, views)            -- feeds popularity tiers
qoe_beacons      → stream to Kafka → aggregates by ISP, device, CDN, player version

Metadata is small: ~100M+ videos × ~2 KB ≈ hundreds of GB, read-heavy and cacheable — PostgreSQL sharded by video_id or Cassandra both work. View counts and QoE beacons are high-volume streams through Kafka into aggregation; the pattern is the same as in Ad Click Aggregation.

Appendix C: Coordination Mechanisms#

Workers run well on Kubernetes with preemptible node pools for the ladder, advanced and backfill classes and on-demand nodes for the fast path.

Appendix D: API Contract & Client Behavior#

GET /videos/{id}/play?device=android&hdr=0&codecs=avc1,vp09,av01
200 {
  "manifest_url": "https://cdn-a.example/v/abc123/master.m3u8?sig=...&exp=6h",
  "alt_manifests": ["https://cdn-b.example/v/abc123/master.m3u8?sig=..."],
  "license_url": null,
  "start_rung": "720p",
  "rung_cap": null,
  "hints_ttl_s": 300
}

Client rules: start at start_rung or lower; on segment failure try the same segment once on the next CDN in alt_manifests before switching down; on 429 honor Retry-After with ±30% jitter; never retry a future live part faster than half the part duration; refresh hints every hints_ttl_s and keep playing with stale hints if the hint call fails.

Caching rules: segments Cache-Control: public, max-age=2592000, immutable; VOD manifests short TTL (~2 s) until LADDER_COMPLETE, then long; live manifests ≤ part duration with blocking reload.

Appendix E: Observability#

E.1 Core Metrics#

# Viewer experience (client beacons)
qoe.startup_time_p50 / p95{device,isp,cdn}
qoe.rebuffer_ratio{device,isp,cdn,player_version}
player.start_failure_rate{reason}
qoe.bitrate_avg_kbps / qoe.bitrate_switches_per_min

# Delivery
cdn.byte_hit_ratio{provider,region}
origin.egress_gbps / origin.ttfb_p99 / origin.shed_requests_total
egress.cost_per_gb{provider,country}

# Pipeline
encode.queue_age_p99{class}
publish.time_to_playable_p50
encode.ladder_completeness
encode.dlq_depth{task_type}

# Economics
storage.bytes{tier} / storage.bytes_per_watch_hour
cost.per_watch_hour (monthly)

E.2 Critical Alerts#

E.3 Debugging "Users Say It Buffers"#

Slice qoe.rebuffer_ratio by ISP → CDN → device → player version → title. A single ISP at evening peak means interconnect congestion (steer, cap); a single CDN means a provider issue (shift weights); a single player version means a regression (halt rollout); a single title means a cold cache or a bad encode. Client-side telemetry is the only place the viewer's experience exists; see Metrics & Alerting Platform.

Appendix F: Scale Evolution#

What you don't build on day one: per-shot encoding, owned edge, hardware encoders, server-side ABR, low-latency live, your own DRM license server. Each one has a trigger in the 3-year path.

Multi-region: metadata and the play API run active-active per region; renditions are replicated to two or more origin regions because they're immutable (replication is simple — see Replication); sources live in one home region plus archive copy. See Multi-Region.

Appendix G: Multi-Tenancy, Fairness & Cost#

• Encode fairness: per-creator quotas on the bulk class so one partner's archive can't starve others; fast path is never quota-limited for ordinary uploads, but upload rate limits apply per account (Rate Limiting).
• Edge fairness: per-product cache quotas (VOD, shorts, live, ads) so one class can't evict another; live gets reserved memory capacity during marquee events.
• Cost allocation: chargeback by bytes delivered and core-hours consumed per product; storage by bytes-month per tier. Teams that can't see their bytes can't reduce them.
• Tradeoff summary: