Technologies referenced in this case study: Redis · Cassandra · Kafka · PostgreSQL · Flink

Related case studies: Distributed Caching · Notification Systems · Database Sharding · Stream Processing · Blob Storage · CDN & Edge Caching

Related patterns: Scaling Reads · Scaling Writes · Degraded Mode · Caching · Data Modeling

How to Use This Case Study#

Organized for interview use first, reference second.

Pure fan-out on write:
t=0:      Account with 600M followers posts a photo
t=+1s:    Fan-out service enqueues 600M inbox inserts
t=+30s:   Fan-out workers saturated; all other posts queue behind it
t=+10min: Normal users' posts take 10+ minutes to reach followers
t=+15min: "Instagram is broken — my friends can't see my post." Support queue spikes.

Hybrid fan-out (push for normal accounts, pull for celebrities):
t=0:      Same post; author flagged as 'pull' (followers > 1M)
t=+50ms:  Post written once to the author's outbox; no fan-out
t=+1s:    Followers opening the app merge celebrity outboxes at read time
t=+1s:    Normal users' fan-out unaffected; p99 delivery stays < 5s

Executive Summary

If you only read one section, read this. Everything else in the case study elaborates the contrast below.

What This Interview Actually Tests#

Feed generation is not a push-vs-pull question. Every candidate has read the Twitter fan-out blog post.

It is a cost-allocation question under power-law skew that tests:

• Whether you clarify what the feed optimizes — completeness, recency, or engagement — before choosing the machinery
• Whether you size read and write amplification with real numbers and put the threshold where the math says
• Whether you separate candidate generation (cheap, precomputed) from ranking (expensive, request-time) and budget latency across them
• Whether you know which correctness properties are hard (deletes, blocks, privacy) and which are soft (a missing post, slightly stale ranking)

The key insight: A feed is a precomputed candidate set plus a request-time ranker. Precompute what's expensive and stable (who follows whom, recent posts from normal accounts). Compute at read time what's cheap or volatile (celebrity posts, ranking, filtering deleted and blocked content). Every design choice is about which side of that line a piece of work lives on — and who pays when it's on the wrong side.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll design the ranked home feed: mostly followed accounts, with a slot for recommended posts. That's where fan-out, ranking latency, and visibility correctness all interact. Chronological is a subset — turn off the ranker. Discovery reuses the ranking stack with a different candidate source."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: These are the public anchors interviewers recognize.

Twitter — Fan-out on Write with Read-Time Merge for Heavy Accounts#

Twitter engineers publicly described (in the widely cited "Timelines at Scale" talk, ~2012–2013) a home timeline built by fanning out tweet IDs into per-user timelines held in Redis, capped at roughly 800 entries, serving on the order of 300K timeline reads per second. Accounts with very large follower counts were not fanned out; their tweets were merged at read time.

Staff insight: The cap (~800 entries) and the celebrity merge are the two numbers that make the design affordable. Name both — the cap bounds memory, the merge bounds write amplification.

Instagram — Ranked Feed Since 2016, Cassandra at Scale#

Instagram moved from a chronological to a ranked feed in 2016 and has since publicly explained that Feed, Stories, Explore, and Reels each use their own ranking systems with different signals. Instagram engineering has also talked publicly about running Apache Cassandra at large scale for feed and inbox-style data, and about Django and PostgreSQL in the core stack.

Staff insight: Instagram's public explanation that each surface has its own ranker is the intent argument made concrete: one candidate/ranking platform, several objective functions, each owned by a product surface.

Facebook — TAO and Multi-Stage News Feed Ranking#

Facebook's TAO (published 2013) is a geographically distributed, read-optimized graph store for the social graph — the "who follows whom / who liked what" data that feeds depend on. Facebook has also described News Feed ranking as scoring thousands of candidate stories per user with models predicting many engagement outcomes, then combining them into a single relevance score.

Staff insight: The social graph and the ranker are separate systems with separate owners. Candidates who merge them into "the feed service" miss the org boundary that makes the system operable.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: Writes land once in the post store and the author's outbox, then flow through Kafka to fan-out workers that push post IDs into active followers' inboxes — unless the author is above the celebrity threshold. Reads assemble candidates from the inbox, the outboxes of followed celebrities, and recommendations; a two-stage ranker orders them within a fixed budget; a visibility filter enforces blocks and deletes against authoritative data; hydration fetches content from caches. Observability watches fan-out lag, feed latency, ranker fallback, and — most importantly — visibility violations.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 15 minutes on the photo upload pipeline and the posts table, then draw fan-out on write, then run out of time before ranking, visibility, or failure. Upload is a blob storage + CDN problem — say so in one sentence and move on.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional scope quickly:

"Users follow accounts and open a home feed showing posts from those accounts, plus some recommended posts, ranked by relevance. They scroll with infinite pagination. Media upload is a separate blob + CDN path I'll treat as solved."

Then the intent questions:

"Two questions change the design. Is the feed chronological or ranked? And how fresh must it be — does a friend's post need to appear within seconds? I'll assume ranked, like Instagram since 2016, with new posts visible to followers within ~10 seconds, and hard guarantees that deleted posts and blocked accounts never appear."

Commit to scale and targets:

"~500M DAU, ~100M posts a day, ~10 feed opens per user per day — about 60K feed requests/sec average, ~150K peak. Feed p99 under 500ms server-side. Fan-out p99 under 10 seconds for normal accounts. Follower counts are power-law: most accounts have hundreds, a few have hundreds of millions."

🎯 Staff Move: Saying "power-law follower distribution" in Phase 1 signals that you already know the celebrity problem is coming and that your design will be driven by skew, not averages.

Phase 2: Core Entities & API (1–2 minutes)#

• Post: post_id (time-sortable, e.g., Snowflake-style 64-bit), author_id, media_refs, caption, created_at, visibility, deleted_at
• Follow edge: (follower_id, followee_id, created_at) — stored both directions
• Inbox entry: (user_id, post_id, author_id, inserted_at) — capped per user
• Author outbox: (author_id, post_id) — recent N posts per author
• Feed session: (user_id, session_id, ranked_ids[], cursor) — short-lived snapshot

API:

POST /v1/posts                    { media_ids[], caption }        → { post_id }
GET  /v1/feed?cursor=&limit=20                                    → { items[], next_cursor }
POST /v1/follows/{user_id}        DELETE /v1/follows/{user_id}
POST /v1/feed/seen                { post_ids[] }                  (batched impressions)

🎯 Staff Move: "Inbox entries hold IDs, not content. That keeps entries ~20 bytes, makes edits and deletes automatically correct at hydration, and means the inbox is a candidate list — never the authority on what's visible."

Phase 3: High-Level Architecture (≤5 minutes)#

Walk it in 60 seconds:

1. Write: post stored once; appended to the author's outbox; post_created event to Kafka.
2. Fan-out: workers page through the author's followers (from the graph store), skip inactive followers and skip entirely if the author is above the threshold, and append the post ID to each inbox.
3. Read: feed service gathers candidates — inbox (~500), outboxes of followed celebrities, recommended posts — about 1,000–2,000 IDs.
4. Rank: light model scores all candidates, heavy model scores the top ~200.
5. Filter & hydrate: drop deleted, blocked, muted, and private content against authoritative data; fetch post and author objects from caches; return 20 items and a cursor.

🎯 Staff Move: "That's the textbook design. The decisions that matter: where the celebrity threshold goes and why, how ranking fits in a 150ms budget and what happens when it can't, and how we guarantee a blocked account never appears when inboxes are eventually consistent. Which first?"

Phase 4: Transition to Depth (1 minute)#

"I'd start with fan-out economics because it sets the cost of the whole system, then the ranking budget and its degraded mode, then visibility correctness — which is the one place I'd refuse to be eventually consistent."

Phase 5: Deep Dives (25–30 minutes)#

Deep dive A: Fan-out economics and the threshold (8–10 min)

"Naive push: 100M posts × ~200 average followers = 20B inbox writes a day, ~230K/sec average, ~700K/sec at peak. Two cuts. First, skip followers inactive for 30+ days — typically 50–70% of follower edges point at people who won't open the app this week. That brings us to ~7–10B writes/day. Second, the celebrity threshold. The top 0.01% of accounts can account for a large share of follower edges. Above ~1M followers, don't push; pull from the outbox at read time."

"Read-side cost of pulling: a user follows on average maybe 5–20 pulled accounts, each outbox is a cached list of the last ~50 post IDs, so it's 5–20 cache reads batched into one multi-get — ~2–5ms. Users who follow 300 celebrities get a capped merge: only the 50 most-engaged-with pulled authors, the rest via recs."

"Victims: pushing to inactive users wastes the fan-out fleet; not pushing means returning users need an inbox rebuild on open — a pull from their top ~200 followees' outboxes, ~50–100ms once, then async repopulation. I accept that because returning users are a small share of opens."

Deep dive B: Ranking within a budget (7–8 min)

"Budget: 500ms p99 server-side. Candidate gen ~30ms, ranking ~150ms, filter + hydration ~50ms, the rest is network and margin. Two stages: a light model — logistic regression or small GBDT on precomputed features — scores 2,000 candidates in ~20ms. The heavy model — a multi-task neural net predicting like, comment, share, dwell, and hide probabilities — scores the top ~200 in ~80–100ms on a GPU/CPU inference fleet. Final score is a weighted combination owned by product. Then diversity rules: no more than 2 consecutive posts from one author, cap recommended slots at ~20–30%."

"Fallback: if the ranker times out at 150ms, serve the candidates sorted by recency with the light-model score as a tiebreak. ranker_fallback_rate above 1% pages the ranking team. Users see a slightly worse feed, not an error."

Deep dive C: Visibility correctness (6–7 min)

"Inboxes are eventually consistent. If Bob blocks Alice, Alice's posts may already sit in Bob's inbox. So visibility is checked at read time: for each candidate, verify author not blocked or muted by viewer (a per-viewer block set, cached, invalidated on change), post not deleted (post cache carries deleted_at, with delete events invalidating immediately), author's account not private-to-viewer. These are hard guarantees. I'd measure them with an audit sampler: re-check 0.1% of served items against the source of truth and page on any violation."

🎯 Staff Move: Every deep dive ends with an owner: "Feed infra owns fan-out and the threshold. The ranking team owns models and the fallback rate. Integrity owns visibility rules and the audit. Product owns the objective weights."

Phase 6: Wrap-Up (2–3 minutes)#

"Summary: hybrid fan-out of post IDs to active followers with a ~1M-follower pull threshold; candidate generation from inbox, celebrity outboxes, and recs; two-stage ranking within 150ms with recency fallback; read-time visibility filters as a hard guarantee; cursor pagination over a per-session snapshot. Next I'd build: real-time feature freshness for the ranker so a post going viral is recognized in seconds; an inbox rebuild path that's cheap enough to drop inboxes for 90-day-inactive users entirely; and an integrity audit dashboard with a zero-tolerance SLO."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Everyone has seen the push/pull answer, so it's a clean test of whether a candidate can go beyond the memorized solution. The skew makes averages lie; the ranking stack adds a latency budget and a machine-learning dependency that fails in new ways; and the product has hard trust-and-safety requirements hiding inside a "just eventually consistent" system. It also has a sharp organizational dimension: graph, ranking, integrity, and feed infra are separate teams, and product owns the objective function.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"Which work are we doing for people who will never see it, and which work are we skipping for people who will?"

Pushing to inactive followers is work for nobody. Ranking 2,000 candidates with the heavy model is work for items that will never be in the top 20. Skipping the visibility check is work skipped for someone who blocked an abuser. The question forces the active-user filter, the two-stage funnel, and the read-time filter in one breath.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Chronological following feed. Completeness is the product promise. Users notice when a post is missing or out of order. Fan-out on write fits well, pagination is trivially stable (sort by time), and ranking is absent. Victims: users who follow many accounts get flooded; creators whose followers are in other time zones lose reach.

Ranked following feed (default). The promise is "the most relevant posts first." Missing a low-relevance post is acceptable, so the inbox can be capped and candidate generation can be lossy. The cost is a ranking stack with a latency budget, feature freshness, and objective-function governance. Victims: ranking infra (inference fleet), creators whose reach depends on a model they can't see, and users when ranking degrades silently.

Discovery feed. No follow graph at all — retrieval from a corpus of billions of posts via embeddings and approximate nearest neighbor search, then ranking. Integrity risk is highest here because content comes from strangers. Victims: integrity team, and users exposed to borderline content if classifiers lag.

2.2 When NOT to Use Fan-out Infrastructure#

🎯 Staff Move: "Until we're past ~10M DAU, I'd build pull-on-read with aggressive caching. Fan-out infrastructure is a cost optimization for read-heavy scale — adopting it early buys operational complexity before it buys anything else."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: Fan-out on Write vs Read#

The Staff default: hybrid, threshold ~1M followers, push only to followers active in the last 30 days, rebuild on return.

When to deviate: chronological products with completeness promises may lower the threshold to ~100K and push to everyone active in 90 days, accepting more cost. Tiny products should pull only.

🎯 Staff Move: "The threshold isn't a constant, it's a cost curve. I'd plot fan-out queue p99 against read-merge p99 as I move the threshold from 100K to 10M and pick the knee. It'll probably land around 1M — and I'd revisit it when follower distributions shift."

3.2 Fault Line 2: Precomputed Feed vs Request-Time Ranking#

The Staff default: precomputed features (author affinity, post engagement velocity) + request-time two-stage ranking. Optionally pre-rank for predicted-next-open users (e.g., those who open at 8am daily) to cut peak load.

When to deviate: at extreme peaks (New Year's midnight), serve lightweight-only ranking for a fraction of traffic.

3.3 Fault Line 3: Freshness/Completeness vs Availability#

The Staff default: everything degrades except visibility. If the block-list service is down, serve only items from authors on the viewer's cached "verified safe" set or return a shorter feed — never unverified content.

3.4 Fault Line 4: Who Gets a Materialized Inbox#

The Staff default: 30-day active users, capped at ~500–1,000 IDs each, stored in a memory-first tier (Redis) for the hottest users and a wide-column store (Cassandra) as backing store. See Distributed Caching.

3.5 Fault Line 5: Pagination Stability vs Ranking Freshness#

The Staff default: on feed open, rank and store the top ~200–500 IDs as a session snapshot (TTL ~30 min) in a cache; cursor = (session_id, offset). When the snapshot is exhausted, generate a new batch excluding seen IDs. Pull-to-refresh creates a new session.

4. Failure Modes & Operational Reality#

4.1 Celebrity Fan-out Storm#

Scenario: An account crosses from 950K to 1.2M followers during a viral moment, but the threshold is evaluated from a daily cached count. It's still classified "push."

t=0:      Author posts 6 times in 10 minutes
t=+1min:  6 × 1.2M = 7.2M inbox writes queued
t=+3min:  Fan-out queue lag p99 rises from 3s to 4min
t=+5min:  Normal users' posts delayed; "my post isn't showing" reports

Detection: fanout_queue_lag_p99 > 30s; fanout_writes_per_author top-K.

Mitigation: Per-author fan-out rate limit — any author exceeding N writes/min is reclassified to pull automatically; separate fan-out queues by author tier so large authors can't starve small ones.

Prevention: Threshold uses real-time follower counts with hysteresis (enter pull at 1M, exit at 800K). Owner: feed infra.

4.2 Ranker Outage — Silent Quality Collapse#

t=0:      Heavy-model inference fleet deploy with bad feature schema
t=+1min:  Heavy model returns NaN scores; feed service treats NaN as 0
t=+1min:  Feed order effectively random among top 200
t=+2hr:   Engagement down 15%; no errors, no latency change

Detection: ranker_score_nan_rate, ranker_score_distribution_shift (KL divergence vs baseline), online engagement metrics per hour.

Mitigation: NaN or out-of-distribution scores trigger fallback to light-model order; roll back model.

Prevention: Shadow scoring new models on live traffic before promotion; schema checks between feature store and model. Owner: ranking team.

4.3 Visibility Violation — Blocked User's Post Shown#

Scenario: The block-list cache has a 10-minute TTL with no invalidation on block events. A harassment victim blocks an account and still sees their posts.

Detection: Audit sampler (0.1% of served items re-verified) → blocked_content_shown_total > 0; user reports.

Mitigation: Invalidate block-list cache on block events (event-driven, <1s); on cache miss, read from source of truth.

Prevention: Visibility is a hard guarantee with a zero-tolerance SLO; changes to filter caching require integrity review. Owner: integrity team (rules), feed infra (enforcement path).

4.4 Inbox Store Hot Shard#

Scenario: Inboxes sharded by user_id, but a bot farm of 50K accounts all following the same hot accounts gets co-located by a bad hash; one shard takes 20× writes.

Detection: inbox_shard_write_qps skew > 5×; shard latency p99.

Mitigation: Consistent hashing with virtual nodes; drop fan-out to accounts flagged as bots.

Prevention: Spam/bot filtering upstream of fan-out. Owner: feed infra + integrity. See Database Sharding.

4.5 Returning-User Rebuild Stampede#

Scenario: A push notification campaign re-engages 20M dormant users in 30 minutes. Each needs an inbox rebuild.

t=0:      Campaign sends
t=+5min:  Rebuild rate 50× baseline; outbox cache misses spike
t=+10min: Feed p99 for all users rises to 2s as rebuilds saturate shared caches

Detection: inbox_rebuild_rate, outbox_cache_miss_rate, feed_latency_p99.

Mitigation: Serve returning users a lightweight feed (top 50 followees' recent posts + popular recs) immediately; rebuild asynchronously with a rate limit.

Prevention: Growth campaigns coordinate with feed infra; staged sends. Owner: feed infra + growth.

4.6 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Knowing the Twitter fan-out story ≠ designing one. Ask where the threshold goes and why.
• Deep ML knowledge ≠ ranking system design. Ask about latency budgets and fallback.
• "We'll use Cassandra" ≠ a data model. Ask for the partition key and row size.
• Mentioning caching ≠ sizing it. Ask how much memory and for which users.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Media upload, transcoding, CDN details — one sentence
• Model architectures — stages and budget only
• Like/comment counter implementation — mention sharded counters, link to Leaderboard & Counting
• Search — separate system

6.4 Follow-Up Questions to Expect#

1. "An account with 600M followers posts. Walk me through it."
2. "How much memory do inboxes need?"
3. "What does the feed look like if ranking is down?"
4. "I blocked someone five seconds ago. Can I see their post?"
5. "How do you paginate a ranked feed without duplicates?"
6. "A user comes back after 6 months. What happens when they open the app?"
7. "How do you know the ranking got worse?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design the Instagram feed."

Drill 2: Size the Fan-out#

Prompt: "How many inbox writes per second do you need to support?"

Drill 3: The Celebrity Posts#

Prompt: "An account with 600M followers posts. Walk me through the next 10 seconds."

Drill 4: Ranking Budget#

Prompt: "You have 2,000 candidates and 150ms. What runs?"

Drill 5: Blocks and Deletes#

Prompt: "I blocked someone 5 seconds ago. Their post is in my inbox. Can I see it?"

Drill 6: Pagination#

Prompt: "User scrolls to page 3. Meanwhile, new posts arrived and ranking changed. What do they see?"

Drill 7: Returning User#

Prompt: "A user returns after 6 months. They have no inbox. What happens?"

Drill 8: Build vs Buy#

Prompt: "We're a mid-size social app (5M DAU). Build this, or use something off the shelf?"

Drill 9: Cost#

Prompt: "Feed infra costs are growing faster than DAU. Where would you look?"

Drill 10: Multi-Region#

Prompt: "Users are global. Where do inboxes live, and what happens when a user in Brazil follows someone in Japan?"

8. Deep Dive Scenarios#

Deep Dive 1: New Year's Eve Peak#

Context: At midnight in several large time zones, feed opens jump 4× and posting jumps 6×. Feed p99 goes from 350ms to 2.5s. Fan-out lag hits 3 minutes. On-call escalates to you.

Questions to Surface First:

• Which stage is slow — candidate gen, ranking, hydration?
• Is fan-out lag caused by a few large authors or overall volume?
• Are degraded modes (lightweight ranking) available and tested?
• Was this peak in the capacity plan?

Typical L5 Approach: Scale out the feed service and fan-out workers. Helps eventually, but autoscaling takes minutes and the peak lasts ~15 minutes per time zone.

Staff Approach: Enable pre-planned peak mode: stage-2 ranking only for 50% of requests (rest get stage-1 order), recs candidates reduced, session snapshots shortened. Fan-out: raise celebrity threshold temporarily down to 100K so more authors are pulled — reducing write load at the cost of slightly heavier reads. Protect hard guarantees: visibility filter stays fully on.

Principal Approach: Peaks are predictable; incidents on them are planning failures. Institute an annual peak calendar (NYE by time zone, major sports finals) with pre-scaling 1 hour ahead, pre-approved degradation levers, and a game day 2 weeks before. Budget peak capacity as an explicit cost line versus planned degradation — and let product choose.

Metrics to Watch: feed_latency_p99 by stage, fanout_queue_lag_p99, ranker_fallback_rate, engagement during peak.

Organizational Follow-up: Peak calendar owned by feed infra; product pre-approves degradation levers.

Ownership Question: Who decides to degrade ranking quality during peaks? Feed infra on-call, using levers product pre-approved — no real-time negotiation at midnight.

Key Takeaway: "Degradation you've rehearsed is a feature. Degradation you invent at midnight is an incident."

What clears the Staff bar:

• Uses pre-planned levers instead of scaling alone
• Moves the threshold as a load-shifting tool
• Keeps hard guarantees untouched

Deep Dive 2: The Silent Relevance Drop#

Context: Over two weeks, time spent in feed fell 6%. No incidents, no errors, no latency changes.

Questions to Surface First:

• What changed: models, features, candidate sources, thresholds, client?
• Is the drop uniform across cohorts or concentrated (new users, a region, an app version)?
• Are feature values fresh?

Typical L5 Approach: Retrain the model on recent data.

Staff Approach: Bisect by cohort and change log. Found: a feature pipeline migration made "engagement velocity" 6 hours stale instead of 1 minute — the ranker kept working but couldn't see trending posts. Fix freshness; add feature_freshness_seconds per feature with alerts; add feature distribution monitoring comparing online vs training.

Principal Approach: Ranking quality needs the same observability rigor as availability. Establish a ranking health SLO (freshness of top features, online/offline feature skew, holdback-group engagement delta) owned by the ranking platform, with a permanent small holdback cohort on a stable model to detect regressions from any source.

Metrics to Watch: feature_freshness_seconds{feature}, online/offline skew, engagement vs holdback.

Organizational Follow-up: Feature pipeline changes require ranking sign-off; migrations shadow-validated.

Ownership Question: Who's paged when a feature goes stale? The feature platform team — they own freshness SLOs; ranking owns the model's sensitivity to it.

Key Takeaway: "A ranker fed stale features fails silently. Monitor the inputs, not just the outputs."

What clears the Staff bar:

• Bisects by cohort and change log
• Finds input staleness, not model quality
• Adds a holdback for permanent detection

Deep Dive 3: Onboarding a Mega-Creator Program#

Context: Product launches a creator program: 50,000 creators will get boosted distribution, many crossing 1M followers within months.

Questions to Surface First:

• How does "boosted distribution" work — ranking boost, or guaranteed candidate inclusion?
• How many creators will cross the pull threshold, and how fast?
• What's the read-side cost of following many pulled authors?

Typical L5 Approach: Add a ranking boost feature for program creators.

Staff Approach: Model the threshold impact: if 5,000 more authors go to pull, typical users follow 3× more pulled authors, so read-merge cost triples. Mitigate with affinity-capped pulls (merge only the top 50 pulled authors by affinity per viewer). Boost belongs in the ranking objective, with an integrity cap on how much boost can override low-quality signals.

Principal Approach: Creator programs change the objective function — that's a governance decision. Define who owns the creator-vs-viewer tradeoff, publish distribution principles, and ensure boost weights are reviewed like policy, with a sunset date.

Metrics to Watch: pulled_authors_per_request_p99, candidate gen latency, creator reach, viewer hide rate.

Organizational Follow-up: Objective changes go through a ranking review with product, integrity, and creator teams.

Ownership Question: Who owns the creator boost weight? Product owns it, bounded by integrity caps that ranking enforces.

Key Takeaway: "Every product program is either a cost curve or an objective change. Find out which before shipping."

What clears the Staff bar:

• Projects infra impact of a product program
• Keeps boosts bounded by integrity
• Separates mechanism from policy

Deep Dive 4: Post-Mortem — Harassment Content Shown After Block#

Context: A public figure blocked a harasser; the harasser's posts appeared in their feed for 40 minutes. It became a press story.

Questions to Surface First:

• Which path served it — inbox candidates, recs, or a pulled outbox?
• Did every candidate source go through the visibility filter?
• What's the block propagation path and latency?

Typical L5 Approach: Found: block cache had a 1-hour TTL. Shorten the TTL.

Staff Approach: The real root cause: the recs candidate source bypassed the visibility filter because it was added later and filtered at its own layer using a stale block snapshot. Fix: single mandatory visibility stage after merge for all candidate sources; event-driven block invalidation; audit sampler; integration test that every candidate source flows through the filter.

Principal Approach: Visibility enforcement must be structurally impossible to bypass: one visibility service, required by the feed framework's type system or pipeline definition, used by every surface. Integrity SLO of zero violations, reported to leadership, with an owner who can block launches.

Metrics to Watch: blocked_content_shown_total by source, block_propagation_seconds_p99, audit coverage.

Organizational Follow-up: New candidate sources require integrity review before launch.

Ownership Question: Who can add a new candidate source? Any team — but only through the pipeline that enforces visibility, and integrity signs off.

Key Takeaway: "Guarantees enforced per-source erode with every new source. Enforce once, after the merge."

What clears the Staff bar:

• Finds the architectural bypass, not just the TTL
• Makes enforcement structural
• Adds measurement

Deep Dive 5: Multi-Region Expansion#

Context: The product is single-region (US). Latency in India and Brazil is 400ms+ on feed opens; leadership wants regional presence.

Questions to Surface First:

• Where is latency spent — network RTT, or server-side stages?
• Any data residency laws for user content?
• Which data must be local (inbox, sessions) vs can be remote (cold posts)?

Typical L5 Approach: Deploy full replicas in each region with bidirectional database replication.

Staff Approach: Home-region users: inbox, session snapshots, and feature caches local. Posts replicated globally via async Kafka mirroring; media via CDN. Social graph replicated read-mostly with writes to a home region. Ranking inference in every region. Fan-out regional. Cross-region lag ~1s — within the 10s delivery target.

Principal Approach: Region design is a platform template with residency rules as inputs. Price each region (~fixed cost of inference fleet + caches) against latency-driven engagement gains measured in an experiment before committing.

Metrics to Watch: feed_latency_p99{region}, cross_region_replication_lag, fan-out lag by region.

Organizational Follow-up: Regional on-call; residency review with legal.

Ownership Question: Who decides whether to open region #4? Leadership, based on a priced proposal from feed infra showing cost vs measured engagement lift.

Key Takeaway: "Put the per-user state near the user; replicate the shared content asynchronously."

What clears the Staff bar:

• Measures before deploying
• Separates per-user from shared data
• Failover via cold-start path

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Clarify ranked vs chronological and hard vs soft correctness before designing
• Size fan-out with real arithmetic and cut it with active-only pushes and a measured threshold
• Explain the celebrity problem and the hot-outbox-key problem that its fix creates
• Budget a two-stage ranking funnel and define its fallback
• Enforce visibility at read time with event-driven invalidation and an audit
• Paginate a ranked feed with session snapshots and seen-sets
• Design degraded modes for each dependency, with "everything degrades except visibility"
• Frame the objective function and ranking platform as org-level decisions (L7)

The Bar for This Question#

Mid-level (L4): Builds a working feed with a follow table and a query over followed users' posts, maybe cached. Fine at small scale.

Senior (L5): Hybrid fan-out with a celebrity exception, Redis inboxes, an ML ranker, cursor pagination. Recognizable, competent design. Gaps: no numbers, no ranking budget, visibility treated as eventually consistent, no degraded modes.

Staff+ (L6): Commits to an intent; sizes fan-out; picks the threshold from cost curves; skips inactive users; builds a two-stage ranker within a budget and a recency fallback; makes visibility a hard read-time guarantee with an audit; names owners for fan-out, ranking, integrity, and the objective. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Push vs Pull Is Solved — The Interview Is About Ranking and Integrity"#

The Staff position: Spend 5 minutes on fan-out with numbers, then move on to ranking budgets, visibility, and degradation.

Why this matters in interviews: Candidates who spend 25 minutes on fan-out are answering 2012's question.

10.2 "Most Fan-out Work Is Wasted"#

The Staff position: Push only to recently active users. It's the biggest cost lever, bigger than the celebrity threshold.

Why this matters in interviews: Shows you optimize for actual usage distribution.

10.3 "The Inbox Should Never Be Trusted"#

The Staff position: Inboxes are caches of candidates. Visibility, deletes, and privacy are enforced at read time against sources of truth — always.

Why this matters in interviews: It's the line between "eventually consistent" as a design and as an excuse.

10.4 "Chronological Is a Degraded Mode, Not a Product Decision"#

The Staff position: Design ranking so that its failure produces a chronological feed automatically.

Why this matters in interviews: Demonstrates graceful degradation as a design property.

10.5 "The Objective Function Is the Most Important — and Least Engineered — Component"#

The Staff position: Weights on engagement predictions encode company values. They need owners, review, and change control like any production config.

Why this matters in interviews: Moves the conversation into ownership, where Staff and Principal signals live.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

At Staff, the feed is a system that assembles and ranks posts. At Principal, the feed is the company's distribution mechanism — it decides which creators grow, which content spreads, and what users spend time on. Feed, Stories, Reels, Explore, and Notifications each want their own ranker, and each team will build its own candidate retrieval, feature pipelines, and visibility checks unless someone decides otherwise. The L7 decisions are: what's the shared ranking and visibility platform, who owns each surface's objective, and how does the company govern changes that shift distribution for millions of creators.

The Org-Level Fault Line#

Shared ranking platform vs per-surface ranking stacks.

The L7 default: shared platform for candidate retrieval, feature store, inference serving, experimentation, and visibility; each surface owns its objective function and its models on that platform.

Cost Model#

Assumptions: ~20 bytes per inbox entry; inbox cap 500; active-only fan-out; stage-2 ranking on ~200 candidates; cloud list prices, rough.

Where money goes: at scale, ranking inference dominates. Each additional candidate scored by the heavy model costs more than every fan-out write for it. The cheapest optimization is often narrowing stage 2.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Feed & Ranking Platform Standard (v1)

Scope: Any surface that selects and orders content for a user (Feed, Stories, Reels, Explore, Notifications ranking).

MUST:

• Pass every candidate, from every source, through the shared visibility service after merge.
• Invalidate visibility caches on block, mute, delete, and privacy events within 1s.
• Define a degraded mode that renders content without the heavy ranker, tested monthly.
• Emit feed_latency_p99, ranker_fallback_rate, feature_freshness_seconds, and visibility_violations_total.
• Register objective-function weights in version-controlled config with an owner and change review.

SHOULD:

• Use the shared feature store and inference platform.
• Maintain a stable holdback cohort (≥0.5%) for regression detection.
• Fan out only to users active within the platform's inactivity horizon.

Exceptions: Ranking platform council, time-bounded, with integrity sign-off.

Success metrics: 0 visibility violations; feed p99 <500ms; fallback rate <1% outside declared peaks; one visibility service company-wide.

What I'd Tell the VP#

"Our feed decides what hundreds of millions of people see and which creators grow, and today four product teams are each building their own version of how that works. I'm proposing one shared platform for ranking and safety checks, while each product keeps control of what it optimizes for. That reduces our largest infrastructure cost — the machine learning that ranks content — by sharing it, and it makes one team accountable for never showing blocked or deleted content anywhere. It needs about 15 engineers for a year. The decisions about what we optimize for become explicit, reviewed, and owned — which protects us when those choices are questioned publicly."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "Ranking owns the model" — true for Feed, silent on the three other surfaces duplicating it.
• "We filter blocked users at read time" — for one surface, not an org-wide guarantee with an owner.
• "Product decides the weights" — without review, versioning, or accountability when distribution shifts.

🧭 Principal Move: "Before we tune the feed ranker, let's decide what the company optimizes for and who signs off when it changes. The infrastructure follows from that — one platform, several objectives, one visibility guarantee."

Appendices

Appendix A: Fan-out Mechanics#

A.1 Push Fan-out Worker#

on post_created(post_id, author_id):
  if follower_count(author_id) > PULL_THRESHOLD: return      # outbox only
  for page in followers(author_id, page_size=5000):
    active = [f for f in page if last_active(f) > now - 30d]
    pipeline: for f in active: LPUSH inbox:{f} post_id ; LTRIM inbox:{f} 0 499
  record fanout_latency(post_id)

Queues are tiered by author follower count (small / medium / large) so large authors can't starve small ones.

A.2 Read-Time Merge#

candidates = inbox[user][:500]
pulled = top_k_by_affinity(followed_pull_authors(user), k=50)
candidates += multiget(outbox[a][:20] for a in pulled)
candidates += recs(user, n=500)
candidates = dedupe(candidates) - seen(user, session)

A.3 Threshold Hysteresis#

Enter pull at ≥1M followers; return to push only below 800K. Evaluate from streaming follower counts, not daily snapshots.

Appendix B: Data Model#

Post IDs are time-sortable (timestamp + shard + sequence), so inbox ordering and chronological fallback need no extra lookup.

Appendix C: Ranking Pipeline#

C.1 Quick Comparison of Candidate Sources#

Appendix D: API Contract and Client Behavior#

• GET /v1/feed returns next_cursor (opaque, encodes session and offset) and new_items_available hint.
• Client prefetches the next page at ~70% scroll; batches impressions every 5–10s.
• Pull-to-refresh starts a new session; client dedupes against the last 200 displayed IDs.
• Retries with jitter; if feed fails, show cached last feed with a banner rather than empty state.

Appendix E: Observability#

E.1 Core Metrics#

feed_latency_ms{p50,p99}{stage}
fanout_queue_lag_seconds{p99}{tier}
ranker_fallback_rate
ranker_score_distribution_shift
feature_freshness_seconds{feature}
blocked_content_shown_total            # must be 0
inbox_rebuild_rate
outbox_hot_key_qps top-K

E.2 Critical Alerts#

E.3 Debugging the Silent Failure#

Relevance drops show up as engagement changes hours later. Check feature freshness, score distributions, and holdback-cohort deltas before touching models.

Appendix F: Scale Evolution#

What You Don't Build on Day One#

• Fan-out infrastructure (pull + cache first)
• Heavy multi-task ranking models (start with a simple scorer)
• Multi-region inboxes
• A custom graph store (a sharded relational table works for a long time)

Appendix G: Multi-Tenancy, Fairness, and Cost#

• Creator fairness: cap per-author share of any viewer's feed; small-creator exploration slots.
• Viewer fairness: heavy followers (5,000+ follows) get affinity-capped candidates — their cost is bounded.
• Cost attribution: cost per 1,000 feed opens, broken into fan-out, ranking, and hydration, reported per surface.