Technologies referenced in this case study: Redis · Kafka · Flink · Cassandra · DynamoDB · PostgreSQL

Related: Scaling Writes · Contention · Data Pipelines · Stream Processing · Real-time Updates · Sharding

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once. Return to individual sections for targeted review.

Without a designed leaderboard (SQL ORDER BY on the scores table):
t=0:       Esports final ends; 4M viewers open the leaderboard
t=+5s:     SELECT ... ORDER BY score DESC LIMIT 100 — 40K QPS, each a 2s index scan
t=+20s:    Primary CPU 100%; score writes from 12K live matches queue up
t=+60s:    "Your rank" query (COUNT(*) WHERE score > mine) times out for everyone
t=+3min:   Prize-eligible top-100 page shows stale data; two players dispute 1st place on stream
t=+2h:     Ops restores; legal asks which ranking was authoritative at 20:00:00

With a designed leaderboard:
t=0:       Same 4M viewers
t=+0s:     Top-100 served from a 1s-TTL cache fed by Redis ZREVRANGE (~0.2ms)
t=+0s:     "Your rank" = ZREVRANK, O(log N), ~20µs server time for 10M members
t=+1s:     Score writes flow through Kafka → single-writer updaters; no lock contention
t=20:00:00 Snapshot of the authoritative board persisted with event offset; prizes paid from snapshot

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What This Interview Actually Tests#

A leaderboard is not a sorted-set question. Everyone knows ZADD.

It is a "what is rank allowed to mean, and who pays when it's wrong" question that tests:

• Whether you separate the top of the board (few rows, many readers, often money attached) from the long tail (millions of rows, each read by one person who wants a rough position)
• Whether you know that exact global rank does not shard — and choose what to give up
• Whether you see write hotness (a viral post, a live match) as a different problem from read hotness
• Whether you define the authoritative snapshot when a prize, payout, or public claim depends on the number

The key insight: Different parts of the same board have different correctness bars. Top-100 must be exact and auditable; rank #4,812,339 can be "top 12%" and nobody is harmed. Staff candidates spend exactness where it has an owner who cares, and approximate everywhere else.

The L5 vs L6 vs L7 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "These three share a sorted order and nothing else. I'll design the real-time competitive board — exact top-K, approximate rank for everyone else, daily and weekly windows — because it forces the sharding and freshness tradeoffs. I'll handle hot counting as a write-path concern inside it, and I'll say explicitly that any payout comes from a batch snapshot, not the live board."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Citing specific production systems demonstrates you've studied operational reality, not textbook designs.

Redis Sorted Sets — The Default Game Leaderboard#

Redis's own documentation and cloud providers' reference architectures (for example, AWS ElastiCache materials) present sorted sets as the canonical leaderboard: ZADD/ZINCRBY to update, ZREVRANGE for top-K, ZREVRANK for "my rank," all O(log N). A single sorted set with tens of millions of members answers rank in microseconds.

Staff insight: The ZSET is the right answer per shard. The interview is about what you do when one board outgrows one shard or one thread — cite the default, then say where it stops.

YouTube's "301 Views" — Counting With a Verification Gate#

For years YouTube public view counts famously appeared to stall around 301 while the system verified that views were legitimate; YouTube has since changed how counts update. The public explanation was that early views were validated before being reflected in the public number.

Staff insight: A public count is a product surface with an integrity bar. Showing a lagging-but-trustworthy number was a deliberate choice of integrity over freshness — the same choice a prize board must make.

Google Cloud Firestore — Distributed Counters#

Firestore's documentation describes a sustained write limit of roughly one write per second per document and recommends distributed counters: split a counter into N shard documents, increment a random shard, and sum on read.

Staff insight: This is the sharded-counter pattern documented by a major vendor as the required workaround for a per-key write ceiling. Every store has such a ceiling — Redis (one thread per shard), DynamoDB (~1,000 WCU/s per partition) — and a Staff answer names it and the split factor.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: Producers are authoritative servers, never clients. Every score is an event in Kafka, partitioned so each board-entity pair has one writer. Flink validates, pre-aggregates 1s of events per key, and assigns events to windows. The serving layer splits by correctness bar: an exact ZSET for the top tier, a histogram for approximate rank of everyone, sharded counters for hot counts, and a 1s cache in front of the top page. Snapshots tie board state to a Kafka offset so any published number can be reproduced — and payout boards come from batch recomputation, not the live ZSET.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates draw ZADD + ZREVRANGE in minute 4 and then spend 20 minutes on API shapes and caching headers. The interviewer's real question — "it has 500 million members now; what is rank?" — arrives at minute 35 with no time left. Get the basics done by minute 10.

Phase 1: Requirements & Framing (2–3 minutes)#

Functional requirements in 30 seconds:

"Update a player's score, show the top N, show my rank and the players around me, for all-time plus daily and weekly windows."

Then intent and non-functionals — the Staff move:

"Is this a live game board, an engagement counter like likes and trending, or a payout ranking where money depends on the order? They have different correctness bars. I'll design the live competitive board, and I'll treat payouts as a batch snapshot problem layered on top."

Commit to numbers:

"Assume 200M registered players, 20M daily active, 200K score updates/s at peak during events, 50K rank reads/s and a top-page spike to 4M concurrent viewers during finals. Top-1,000 exact within 2 seconds of the event. Everyone else gets rank as a percentile within ±0.1%. Scores come only from game servers."

🎯 Staff Move: "Exact top, approximate tail" is the load-bearing sentence. It decides the sharding strategy, the memory budget, and the failure behavior before you draw anything.

Phase 2: Core Entities & API (1–2 minutes)#

• ScoreEvent: event_id, board_id, entity_id, delta or new_score, source_match_id, event_time
• Board: board_id, aggregation (max, sum, latest), windows (all, daily, weekly), tiebreak (earliest), exact_top_k
• RankView: rank (exact or null), percentile, score, neighbors[]

POST /boards/{id}/events        (internal, game servers only)  { event_id, entity_id, delta, event_time }
GET  /boards/{id}/top?window=weekly&n=100            → [{ rank, entity_id, score }]
GET  /boards/{id}/me?window=weekly                   → { rank | null, percentile, score }
GET  /boards/{id}/around/{entity}?window=weekly&k=5  → neighbors (exact only in top tier)

🎯 Staff Move: "The response shape encodes the correctness contract: rank is present only when it's exact; otherwise the client shows percentile. The UI can't accidentally promise precision the backend doesn't have."

Phase 3: High-Level Architecture (≤5 minutes)#

Walk the flow in 90 seconds:

1. Game servers publish ScoreEvents to Kafka, keyed by (board_id, entity_id) — per-entity ordering, one writer per entity
2. Flink dedupes by event_id, assigns events to windows (all-time, 2026-09-30, 2026-W40), and folds 1s of events per entity into one update
3. For each window: update the entity's score in the canonical score store, update the histogram (decrement old bucket, increment new), and ZADD into the top-tier ZSET if the score is above the tier threshold
4. Top page: ZREVRANGE 0 99 behind a 1s cache
5. "My rank": if my score ≥ tier threshold, ZREVRANK (exact); else histogram lookup (approximate percentile)

Key points to state:

1. Scores are events — replayable, auditable, idempotent
2. Single writer per entity — no read-modify-write races on max aggregation
3. Two serving structures by correctness bar — exact ZSET for the top, histogram for the tail
4. Windows are keys, not resets
5. Top page is cached; rank is not — fan-in on the top, fan-out on rank

🎯 Staff Move: Then say: "This works up to maybe 50M members per board on a single ZSET. The interesting part is what breaks at 500M, what happens when one entity gets 50K events a second, and what happens at the window boundary."

Phase 4: Transition to Depth (1 minute)#

"Three places worth going deep: what rank means when the board doesn't fit on one shard, hot writes on a single entity or counter, and window boundaries plus late events. Which is most interesting to you?"

Default if no preference: sharding and rank semantics — it's the question every leaderboard interview eventually reaches.

Phase 5: Deep Dives (25–30 minutes)#

Deep dive A: Rank at 500M members (8–10 min)

"At ~100 bytes a member, 500M is ~50 GB — past what I want in one Redis shard, and one shard's thread would serve every rank read. Options: hash-shard the ZSET 64 ways — top-K becomes a merge of 64 per-shard top-Ks, fine, but rank becomes Σ ZCOUNT(score, +inf) across 64 shards on every call, 64× the read load. Score-range sharding — shard 0 holds scores ≥ 10,000, shard 1 holds 5,000–9,999 — keeps rank at 'offset of higher shards + local rank', but score distributions shift over a season, so ranges need rebalancing, and the top shard is the hot one. Or the one I'd pick: keep only the top tier exact — the top 100K in one ZSET, ~10 MB — and serve everyone else from a 10K-bucket histogram. Rank error below the tier is bounded by bucket population; with adaptive bucket edges I size buckets so none holds more than 0.1% of members."

Deep dive B: Hot writes (6–8 min)

"Two kinds of hot. A hot entity — a streamer's team gets 50K score events/s during a raid — is handled by Flink pre-aggregation: 1s windows per key collapse 50K events into one ZINCRBY. A hot counter with no natural owner — likes on a viral post at 2M/min — gets both pre-aggregation and a 16-way split: likes:{post}:{0..15}, writes pick hash(event_id) % 16, reads MGET 16 keys. Single-key ceiling is about 100K ops/s on one Redis thread, and pre-aggregation drops us to ~1 write/s/key anyway, so the split matters mainly for stores with low per-key limits like DynamoDB."

Deep dive C: Windows and late events (5–7 min)

"Weekly board = key lb:{board}:2026-W40. At the boundary, nothing resets; new events go to W41, and W40 gets a TTL of 7 days after close. The hard case: a match that ended at 23:59:58 whose event arrives at 00:00:03. I assign windows by event_time from the game server, not arrival time, and hold the window open for a 5-minute allowed lateness. At close + 5 min, I snapshot W40 with its Kafka offset — that snapshot is the official result. Events later than that go to a dead-letter topic for manual review, because silently changing a published final ranking is worse than excluding one late event."

🎯 Staff Move: End each dive with who owns it: "Game-design owns the window definition and lateness; trust-and-safety owns quarantine rules; the platform owns update lag."

Phase 6: Wrap-Up (2–3 minutes)#

"Summary: scores as events, one writer per entity, exact top tier in a ZSET, histogram for approximate rank, windows as keys with snapshots at close, pre-aggregation for hot writes, and a 1s cache on the top page. Payouts come from batch recomputation off the event log, never the live board. Next I'd build friends leaderboards — that's a different problem, a small ZSET per user or on-read computation over ~200 friends — and I'd deliberately not build exact global rank for the tail unless someone can name who needs it."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

The leaderboard looks like the easiest question in the catalog because the textbook answer is one data structure. That's exactly why it's used: it reveals whether a candidate stops at the data structure or asks what the numbers are for.

Real leaderboards are fought over. Players screenshot their rank. Streamers argue on air about who was first. Creator programs pay out based on view rankings. Sales contests pay bonuses. The moment a number is public or paid, it acquires stakeholders — and each stakeholder has a different correctness bar. The Staff skill being tested is allocating exactness: spending memory, latency, and engineering effort on precision where someone is harmed by imprecision, and nowhere else.

The counting half of the problem tests the other classic Staff skill — recognizing that a single key is a single-threaded bottleneck no matter how large the cluster, and designing the write path around key skew rather than average load.

1.2 The L5 vs L6 Contrast — Visual#

The L5 path is right for a 10M-member board. It stalls at step 5 because "exact rank for everyone" was assumed rather than chosen, so sharding breaks a promise that was never priced.

1.3 The Staff Question That Cuts Through Everything#

"Who is harmed if this rank is off by 1% — and do they get paid based on it?"

Asking it out loud forces:

• Correctness allocation: exact for the top, approximate for the tail
• Freshness: real-time for display, snapshot for anything consequential
• Ownership: who adjudicates disputes, and from which recorded state

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Intent 1: Real-time competitive board. Games, fitness challenges, coding contests. Millions of members, frequent updates, heavy reads of the top page and of "my rank." The correctness bar is exact at the top, fast everywhere. Aggregation is usually max (best score) or sum (points). Windows matter: daily and weekly boards drive re-engagement. The hard problems: rank at scale, window boundaries, and cheating.

Intent 2: Engagement counting & trending. Likes, views, shares, "top posts this hour." Write volume dwarfs read volume at the key level, key popularity is power-law distributed, and the "board" (trending) is computed over an unbounded key space. The correctness bar is monotonic and eventually accurate for counts, approximate for trending. The hard problems: hot keys, pre-aggregation, heavy-hitter detection with bounded memory, and counts that must never visibly go down.

Intent 3: Money-bearing ranking. Creator payouts, sales contests, tournament prize pools, affiliate rankings. Lower volume, extreme scrutiny. The correctness bar is reproducible and auditable: given the event log up to cutoff T, anyone can recompute the ranking and get the same answer. The hard problems: cutoff semantics, fraud filtering that's applied consistently, and a dispute process.

🎯 Staff Move: "The live board is for engagement; the payout board is for accounting. I'll never pay anyone from a Redis ZSET — the payout job recomputes from the event log in batch with fraud filters applied, and publishes a signed snapshot. The live board can be wrong for 2 seconds; the payout board can't be wrong at all."

2.2 When NOT to Build a Real-Time Leaderboard#

"If nobody refreshes the page expecting it to change within seconds, it isn't a real-time leaderboard, and I shouldn't pay for one."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Fault Lines#

3.1 Fault Line 1: Exact vs Approximate Rank#

The tension: Exact rank for every member is what users seem to want and what the naive structure provides — until the board outgrows one shard, at which point exact rank for the tail costs a scatter-gather per read.

Staff default: Exact top tier (top 10K–100K) in one ZSET + histogram for everyone. Show #rank inside the tier and Top X% outside it.

When to deviate: Boards under ~50M members — keep one exact ZSET; it's simpler and fits. Boards where every member's exact position matters contractually (a 500-person sales contest) — exact, but that's small.

"Exact rank at position 4 million is a number nobody screenshots and nobody gets paid for. I'll spend exactness where there's a stakeholder."

3.2 Fault Line 2: Freshness vs Write Amplification#

The tension: Applying every event immediately gives the freshest board and the highest write load. Batching reduces load and introduces lag.

Staff default: 1s pre-aggregation in Flink for all boards; 5s for engagement counters; batch for payouts.

"One second of lag buys a 1,000× reduction on hot keys. That's the best trade in this design."

3.3 Fault Line 3: One Sorted Set vs Sharded Board#

The tension: How do you split a board that doesn't fit, when rank is a global property?

Staff default: Top tier + histogram for global; per-league boards (many small exact ZSETs) where game design allows — leagues of 50–100 players are how many games sidestep global rank entirely, and they're better for engagement.

3.4 Fault Line 4: Window Semantics#

The tension: "Weekly leaderboard" hides four decisions: tumbling or sliding, which time zone, event time or arrival time, and what happens to late events.

"A sliding window has no final result, so nobody can win it. If a board ever awards anything, it's tumbling."

3.5 Fault Line 5: Trust vs Throughput in Score Ingestion#

The tension: Every validation step adds latency and complexity; every skipped step is a door for cheaters and double-counting.

Staff default: Server-authoritative events with event_id dedupe, per-event plausibility checks (score delta ≤ max possible per match), statistical outlier quarantine (e.g., > 6σ above the player's history or above the current #1), and a retroactive removal path that updates the board and the snapshots.

"The public top-10 is an attack surface. I'll hold a score out of the top tier for review before I'll let a cheater sit at #1 on a stream for twenty minutes."

4. Failure Modes & Operational Reality#

4.1 The Hot Key That Stalled a Shard#

t=0:       Celebrity posts; likes counter likes:{post_9} lives on Redis shard 14
t=+30s:    38K INCR/s on one key; shard 14's single thread at 95% CPU
t=+45s:    Other 1.2M keys on shard 14 see p99 latency rise from 0.3ms to 40ms
t=+60s:    Unrelated leaderboard reads on shard 14 time out at 50ms; API returns 5xx for 3% of rank calls
t=+2min:   Client retries double load on shard 14
t=+4min:   On-call identifies hot key via redis-cli --hotkeys / key-level sampling
t=+6min:   Enable pre-aggregation path for the post; INCR rate → 1/s; shard recovers

Detection: redis.shard_cpu{shard} > 80%, hot_key.ops_per_sec top-K sampling, counter.write_rate{key} > 10K/s, p99 skew between shards.

Blast radius: Every key co-located on the hot shard — which is why the hot key harms other teams.

Mitigation: Route all counter writes through the stream pre-aggregator by default; auto-split keys that cross 5K writes/s into 16 sub-keys.

Prevention: Never allow direct INCR from request paths for public counters; client-side retry budgets.

Owner: Counter platform team; the posting product's team for retry behavior.

4.2 The Window-Boundary Herd#

A daily board naively resets at 00:00 UTC by a cron job that runs DEL on 40K per-region/per-mode keys, while the client app refreshes every player's rank at midnight to show "new day."

t=00:00:00  Reset job issues DEL on 40K ZSETs (some with 5M members)
t=+0.2s     DEL of a 5M-member ZSET blocks the Redis thread ~2–3s (synchronous free)
t=+1s       20M clients refresh "my rank" simultaneously; half hit blocked shards
t=+3s       Events for the new day arrive; some land before DEL, some after → new-day scores lost

Fix: New key per window (lb:{board}:{date}), no deletes; old keys get EXPIRE (Redis frees large values lazily with lazyfree-lazy-expire yes / UNLINK); clients refresh with jitter (0–60s); first read of the new window shows "Day started — play to rank."

Detection: redis.blocked_ms, board.events_dropped_total{reason="window_race"}, rank-read p99 at boundary.

Owner: Platform (key scheme), client team (refresh jitter).

4.3 Double-Counted Scores After a Consumer Rebalance#

sum boards (total points this week) are only correct if each event is applied once. A Flink job restart replays 90 seconds of events from the last checkpoint; if the sink applies ZINCRBY non-transactionally, those 90 seconds are counted twice.

Detection: Reconciliation job: recompute a sample of 10K players' weekly sums from the event log every 15 min; alert if > 0.1% mismatch. Metric board.reconcile_mismatch_ratio.

Mitigation: Idempotent sinks — write absolute values (Flink holds the per-entity sum in state; sink does ZADD with the absolute score, which is idempotent) rather than increments. For max boards, ZADD … GT is naturally idempotent.

Owner: Platform streaming team.

4.4 Cheater at #1 During a Live Final#

An exploit lets a player submit a score 40× the plausible max. It passes the per-match check (the check used a stale max-score constant from a previous season). The player sits at #1 on stream for 22 minutes.

Detection: board.top1_score_jump_ratio (new #1 score / previous #1 > 1.5× pages trust & safety), outlier quarantine hit rate.

Mitigation: Quarantine new entries into the top 100 whose score exceeds the previous #1 by more than 20% until reviewed (auto-release after 10 min if no action). Remove retroactively from ZSET, histogram, and snapshots.

Owner: Trust & safety (rules), game team (max-score constants per season), platform (quarantine mechanism).

4.5 Silent Staleness — The Board Stopped Moving#

The Flink job's Kafka consumer lags after a partition reassignment. The board is still served, still looks plausible, and is 40 minutes stale. Nobody notices until a player tweets that their rank hasn't changed all evening.

Detection: board.update_lag_seconds = now − event_time of latest applied event; alert at > 30s p99 for 5 min. Also synthetic canary: a test entity scores every 10s on a hidden board; alert if its rank isn't updated within 5s.

Owner: Platform.

4.6 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Skip-list internals ≠ leaderboard design. Knowing why ZSETs are O(log N) says nothing about what to do when one doesn't fit.
• Naming Count-Min Sketch ≠ knowing when to use it. Sketches are for unbounded key spaces (trending), not for a player's own score.
• Elaborate caching layers ≠ read scaling judgment. The top page needs one 1s cache; per-user rank must not be cached per user at 20M users.
• Kafka + Flink everywhere ≠ Staff. For a 1M-member board with 100 writes/s, Postgres and a minute-level refresh is the mature answer.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

6.4 Follow-Up Questions to Expect#

1. "How do you encode tie-breaking in a double score without losing precision?"
2. "How do you rebuild the top-tier ZSET after losing a Redis primary and its replica?"
3. "How would you implement a sliding 24-hour board?"
4. "How do you show 'players around me' when I'm outside the exact tier?"
5. "What does it cost to keep daily boards for 365 days?"
6. "How do you remove a banned cheater from all historical boards?"
7. "How do you run a board across three regions with one global ranking?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design a leaderboard for our game."

Drill 2: Core Mechanic — Ties#

Prompt: "Two players both have 9,800 points. Who's first, and how do you implement it in a sorted set?"

Drill 3: "Top Tier + Histogram" — Make It Concrete#

Prompt: "You said approximate rank from a histogram. Walk me through the data structure and the error."

Drill 4: Dependency Down — Top-Tier Redis Lost#

Prompt: "The Redis primary holding the top-tier ZSET dies and the replica was 30 seconds behind."

Drill 5: Hot Key — Viral Counter#

Prompt: "A post gets 2M likes per minute. Your counter is one Redis key."

Drill 6: Multi-Tenant — Boards as a Platform#

Prompt: "Now 40 games in the studio want leaderboards. One team's event generates 10× everyone else's traffic."

Drill 7: Build vs Buy#

Prompt: "Why not use a managed game-backend leaderboard service?"

Drill 8: Policy Change Without Outage — Changing Scoring Rules Mid-Season#

Prompt: "Game design wants to change a scoring rule halfway through the season."

Drill 9: Cost#

Prompt: "We keep daily, weekly, and all-time boards for 20 regions and 5 modes. What does storage cost?"

Drill 10: Multi-Region#

Prompt: "Players are in three regions. We want one global board."

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Incident — The World Final#

Context: During the championship final, 4.2M concurrent viewers open the leaderboard. Top-page p99 is 3s and rising; score updates are lagging 90s. Game servers for 12K concurrent matches keep writing.

Questions to Surface First:

• Is the pressure on reads (top page, rank) or writes (score updates)?
• Is the top-page cache effective, or are misses stampeding the top-tier shard?
• Which shard(s) are hot, and what else lives on them?

Typical L5 Approach: Adds read replicas and scales Redis. Replica sync adds more load on the stressed primary; the fix takes 20 minutes.

Staff Approach: Checks cache hit rate first: finds the 1s TTL expires simultaneously on 40 API nodes, each missing into the shard (40 ZREVRANGE/s — fine), but the "around me" endpoint is uncached and 4M viewers call it on page load. Disables "around me" via feature flag for non-participants (viewers aren't on the board), keeps it for players. Load drops 90% in 1 minute.

Principal Approach: Separates viewer traffic from player traffic as product surfaces with different SLOs — a spectator view served from CDN-cached snapshots every 2s — and makes tournament events a capacity-calendar item with a pre-event load test.

Key Takeaway: "At peak, your largest audience is often not your users of record. Find who's reading and whether they need what they're reading."

What clears the Staff bar:

• Finds the uncached endpoint rather than scaling everything
• Uses a pre-agreed shed order
• Converts the incident into a product surface split

Deep Dive 2: Silent Failure — Weekly Sums Drifting Up#

Context: A player reports that their weekly points are 12% higher than their match history adds up to. Spot checks show about 3% of players are over-counted, all this week.

Questions to Surface First:

• When did the drift start, and does it correlate with a deploy or a consumer restart?
• Does the sink use increments or absolute values?
• Is the reconciliation job running, and why didn't it alert?

Typical L5 Approach: Fixes the affected players' scores with a script.

Staff Approach: Correlates the drift with a Flink restart on Tuesday that replayed 4 minutes of events into a ZINCRBY sink added by a new board type that bypassed the standard absolute-value sink. Recomputes the week's sums from the event log, rewrites the board, and makes the absolute-value sink mandatory (increment sinks fail the pipeline lint). The reconciliation job existed but sampled only all-time boards — expands it to every board type.

Principal Approach: Makes idempotent sinks a platform invariant enforced in CI and notes the governance gap: a new board type shipped without the platform's review. Adds board-type registration with required reconciliation coverage.

Key Takeaway: "At-least-once delivery plus increment writes equals silent inflation. Write absolute values."

What clears the Staff bar:

• Links drift to replay semantics immediately
• Fixes the invariant, not the rows
• Extends detection coverage to where it was missing

Deep Dive 3: Large-Customer Onboarding — The Mega-Title Launch#

Context: A new title expects 80M players in the first month and a single global board — 4× the current largest board. Game design insists on "exact rank for everyone, it's core to the fantasy."

Questions to Surface First:

• What does exact rank at #52,000,000 give a player that "Top 65%" doesn't?
• Would leagues (divisions of 100) meet the design goal?
• Are prizes attached?

Typical L5 Approach: Provisions a large Redis node for an 80M-member ZSET (~8–10 GB) and hopes the thread holds.

Staff Approach: Shows the numbers: 80M-member ZSET fits memory but all rank reads (projected 150K/s at launch) hit one thread — over the ceiling. Proposes exact top 100K + histogram, plus exact rank inside leagues of 100 (where most players' competitive attention actually sits). Offers a UX test comparing "#52,114,209" vs "Top 65% · #3 in your league."

Principal Approach: Makes league-based ranking the studio's default pattern for new titles, backed by engagement data, so the platform never needs single boards beyond ~50M members.

Key Takeaway: "The cheapest scaling fix is often a product change that users prefer anyway."

What clears the Staff bar:

• Quantifies the thread ceiling, not just memory
• Reframes the requirement with UX evidence
• Attaches cost to the product choice

Deep Dive 4: Post-Mortem — The Wrong Winner Got Paid#

Context: A $50K sales-contest prize was paid to the rep shown #1 on the live board at the deadline. Two days later, refund events arrived that dropped them to #3. The rightful winner escalated to the VP.

Questions to Surface First:

• What were the published contest rules on cutoff and refunds?
• Was the payout taken from the live board or a snapshot?
• Is there a defined dispute window?

Typical L5 Approach: Adds refund handling to the live board so ranks update correctly.

Staff Approach: Identifies the real bug: payouts read the live board, and the contest had no rule for events that arrive after the deadline but refer to activity before it. Proposes: payout board computed in batch at deadline + N days (refund window, e.g., 14 days), from the event log, with a published provisional ranking at the deadline and a final ranking after the window. Snapshot signed and stored with the offset.

Principal Approach: Establishes a standard for any money-bearing ranking: provisional vs final results, a dispute window, a named adjudicator (sales ops, not engineering), and legal review of contest terms. The platform only publishes; it never decides.

Key Takeaway: "A live board is a display. A payout needs a cutoff, a settling window, and an owner who isn't the database."

What clears the Staff bar:

• Finds the missing rule, not just the missing code
• Separates provisional and final results
• Moves adjudication to the business owner

Deep Dive 5: Multi-Region Expansion — Global Board With Residency#

Context: The game launches in the EU and Asia. Player identifiers are personal data; EU counsel says EU player IDs shouldn't be replicated to the US. Product wants a single global top-100.

Questions to Surface First:

• What must the global board show — display names, IDs, both?
• Can we use pseudonymous board IDs?
• Is global rank needed, or just global top-100 + regional rank?

Typical L5 Approach: Replicates all score events to a US region to compute the board.

Staff Approach: Computes regional top tiers and histograms in-region. The global merger receives only pseudonymous board IDs (hmac(region_key, player_id)) and scores for each region's top 1,000, plus histogram counts (aggregates, not personal data). Global top-100 = merge of 3 × 1,000. Display names are resolved in the viewer's request by calling the owning region's profile service.

Principal Approach: Takes the "aggregate-only crosses borders" pattern into the org's data-residency standard so every future global feature (trending, counts) reuses it.

Key Takeaway: "Mergeable aggregates — top-K lists and histograms — let rankings go global without moving personal data."

What clears the Staff bar:

• Uses mergeability as a privacy tool
• Explicit partial-board degradation
• Routes the residency decision to its owner

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Separate competitive boards, engagement counting, and money-bearing rankings, and commit to one in the first 2 minutes
• Allocate exactness: exact top tier, approximate tail with a numeric error bound
• Explain why hash-sharding breaks global rank and choose between tier+histogram, score-range shards, and leagues
• Quantify the single-key ceiling and fix hot writes with pre-aggregation and split counters
• Implement windows as keys with event-time assignment, allowed lateness, and snapshots at close
• Make stream sinks idempotent by writing absolute values
• Define the trust boundary (server-authoritative scores, quarantine) and the payout boundary (batch snapshot, dispute process)

The Bar for This Question#

Mid-level (L4): SQL table with an index; ORDER BY for top-K; COUNT(*) for rank. Works at 100K members.

Senior (L5): Redis sorted sets, ZADD/ZREVRANK, caching the top page, sharding "by user." Competent and fast at 10M members. Stalls on "rank across shards," "one viral key," and "which number do we pay from."

Staff+ (L6): Treats rank as a contract with different bars for different parts of the board, designs the write path around key skew, makes windows and replays safe by construction, and names owners for rules, cheating, and payouts. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 Nobody Needs Exact Rank Below the First Page#

The Staff position: Exact rank in the tail is a cost with no stakeholder. "Top 37%" is more motivating than "#74,112,093" and costs O(1).

Why this matters in interviews: Proposing it — with an error bound — is the clearest signal that you allocate correctness deliberately.

10.2 Global Leaderboards Are a Product Anti-Pattern at Scale#

The Staff position: At 100M players, a global board is demotivating for 99.99% of them. Leagues of ~100 are both better engagement and trivially scalable — many small exact ZSETs spread naturally across a cluster.

Why this matters in interviews: Changing the requirement is sometimes the best engineering move. Say it with data, and offer the global top tier as well.

10.3 Counters Should Never Be Increments at the Store#

The Staff position: The stream owns the count; the store receives absolute values. Increments at the store are how counters drift silently.

Why this matters in interviews: It shows you've reasoned about at-least-once delivery end to end.

10.4 The Live Board Should Never Pay Anyone#

The Staff position: Real-time and reproducible are different products. Money flows from batch-recomputed, snapshot-backed, provisional-then-final results.

Why this matters in interviews: It separates display correctness from financial correctness — a distinction L5 answers almost never make.

10.5 Count-Min Sketch Is Overused in Interviews#

The Staff position: Sketches are for heavy-hitter detection over unbounded key spaces (trending). A like count for a known post should be exact and cheap with pre-aggregation. Reaching for a sketch where a HINCRBY batch works is complexity theater.

Why this matters in interviews: Picking the simplest structure that meets the bar is the Staff instinct.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

At Staff level, a leaderboard is one system with an exactness budget. At Principal level, it's one instance of a pattern the whole company reinvents: like counts, view counts, trending, seller rankings, creator funds, sales contests, gamified fitness streaks. Each team builds its own counter on its own store, discovers the hot-key ceiling during its first viral event, and ships its own replay bug. Two of them pay people from live numbers. The L7 move is to recognize counting and ranking as a platform primitive with declared correctness tiers, and to separate "numbers we display" from "numbers we pay on" as an org-wide rule.

The Org-Level Fault Line#

A shared counting & ranking platform vs per-product implementations.

The Principal position: Shared platform for counting and display rankings; money-bearing rankings stay in the owning business domain but must consume the platform's event log and snapshot format.

Cost Model#

Assumptions: managed Redis ~$0.02/GB-hour-equivalent pricing tier, shared Kafka/Flink clusters with allocated share, engineer fully loaded ~$250K/year.

Exact-rank-for-everyone at Medium scale roughly triples Redis cost (full-member ZSETs per window, scatter-gather replicas) — ~$60–100K/month extra. That's the number to show when product asks for it.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Counting and Ranking Standard (v1)

Scope: Any user-visible count or ranking, and any ranking used for compensation or prizes.

MUST:

1. Source all counts and scores from an append-only event log with unique event_ids.
2. Write stores with idempotent operations (absolute values or set membership); no store-side increments from at-least-once consumers.
3. Declare a correctness tier per surface: exact, exact-top-k + approximate, or approximate, with the error bound shown in the API contract.
4. Rankings used for money MUST be computed from a final snapshot after a declared settling window, with a named business adjudicator.

SHOULD:

1. Route counter writes through the platform's pre-aggregation path.
2. Run reconciliation against the event log for every board type at least hourly.
3. Use tumbling UTC windows unless product justifies otherwise.

Exceptions: Approved by the counting platform tech lead; money-bearing exceptions also require finance sign-off.

Success metrics: zero payouts from live boards; reconcile_mismatch_ratio < 0.01% on all boards; no hot-key incidents affecting co-located tenants for two consecutive quarters.

What I'd Tell the VP#

Our leaderboards and counters work, but every team builds its own, and we've had three incidents this year from the same two bugs: viral traffic overwhelming a single counter and replays inflating scores. Last quarter we paid a contest prize to the wrong person because the payout read a live number. I'm proposing one shared counting platform and one rule: numbers we display can be approximate and fast, but numbers we pay on come from a settled, reproducible snapshot owned by the business. It's a team of four and roughly $40K/month in infrastructure, and it removes an entire class of incidents plus a real financial and legal exposure.

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll build a great leaderboard service for this game" — no view of the other five teams building counters.
• "Payouts use a snapshot" — correct, but no settling window, adjudicator, or contest-terms ownership.
• "We'll shard by region for multi-region" — no residency analysis, no mergeable-aggregate reasoning.

Appendices

Appendix A: Mechanics in Depth#

A.1 ZSET Operations for a max Board#

# Update best score (Redis 6.2+): only raise, never lower
ZADD lb:{board}:2026-W40 GT <composite_score> <player_id>
# Top 100
ZREVRANGE lb:{board}:2026-W40 0 99 WITHSCORES
# My rank (0-based)
ZREVRANK lb:{board}:2026-W40 <player_id>
# Around me (±5)
r = ZREVRANK ...; ZREVRANGE lb:... max(0,r-5) r+5 WITHSCORES
# Trim top tier to 100K members
ZREMRANGEBYRANK lb:{board}:2026-W40 0 -100001

ZADD GT is idempotent for max boards — replays can't lower or inflate.

A.2 Score Histogram#

state: edges[0..B], counts[0..B-1]           # B = 10,000
on_update(old, new):
  if old != null: counts[bucket(old)] -= 1
  counts[bucket(new)] += 1
approx_rank(s):
  b = bucket(s)
  higher = sum(counts[b+1..B-1])             # maintain suffix sums for O(1)
  frac = (edges[b+1] - s) / (edges[b+1] - edges[b])
  return higher + frac * counts[b]
rebalance hourly: split buckets with counts > 0.1% of N; merge adjacent sparse buckets

A.3 Sharded Counter#

write(key, n):  INCRBY key:{hash(event_id) % S} n          # S = 16 for hot keys
read(key):      sum(MGET key:0 .. key:S-1)                 # or cached sum, 1s TTL
promote: if write_rate(key) > 5K/s for 30s → S = 16; demote when < 500/s for 1h

A.4 Heavy Hitters (Trending)#

Count-Min Sketch (w=2,000, d=5, ~40 KB) per window gives count estimates with error ≤ 0.14% of window total at 99.3% confidence; a min-heap of the top 1,000 candidates is updated when an item's estimate exceeds the heap minimum. Windows of 5 minutes merged into an hour by summing sketches cell-wise. Space-Saving with m = 10 × K counters is the alternative when you need guaranteed top-K membership with bounded error.

Appendix B: Keys and Data Model#

Window IDs: all, d2026-09-30, w2026-W40, s2026-S3 (season).

Appendix C: Sharding Strategies — Quick Comparison#

Appendix D: API Contract#

• rank is returned only when exact; otherwise null with percentile and percentile_error (e.g., 0.1).
• Top page: Cache-Control: max-age=1, stale-while-revalidate=5.
• as_of timestamp and board_version (Kafka offset) on every response — clients can detect staleness; support can reproduce disputes.
• Payout endpoints return status: provisional | final; only final is valid for payment.

Appendix E: Observability#

board.update_lag_seconds{board}           # event_time → visible; SLO p99 < 2s for top tier
board.reconcile_mismatch_ratio{board_type}
redis.shard_cpu{shard}, hot_key.ops_per_sec{key}
rank.approx_error_estimate{board}
board.top1_score_jump_ratio{board}
canary.rank_update_seconds                # synthetic player on a hidden board
cache.hit_ratio{endpoint}

Appendix F: Scale Evolution#

What you don't build on day one: sliding windows, sketches for trending, multi-region merge, exact rank for the tail, real-time push of rank changes.

Appendix G: Multi-Tenancy, Fairness, and Cost#

• Per-tenant ingestion quotas (events/s) and board count limits; 429 to publishers with buffering.
• Hash-tag placement groups each tenant's keys; a hot tenant heats only its slot group.
• Retention tiers: 7 days hot in Redis, snapshots in object storage for 13 months, event log in OLAP.
• Chargeback: per 1M events ingested, per GB-month of hot board storage, per board-window retained.
• Fairness signal: ingest.share{tenant} > 50% of cluster for 10 min → notify tenant and platform.