Technologies referenced in this case study: Kafka · Flink · Time-Series Databases · Redis · Cassandra · DynamoDB

Related case studies: Stream Processing · Idempotency · Leaderboards & Counting · Payment Processing · Rate Limiting

How to Use This Case Study#

Organized for interview use first, reference second. Stream-engine internals (checkpoint barriers, watermark propagation, state backends) live in Stream Processing and Flink; this case study is about counting money correctly while also counting fast.

Without a correctness design:
t=0:      Flink job restarts after a TaskManager OOM
t=+40s:   Job resumes from last checkpoint (taken 4 min ago)
t=+40s:   Sink is append-only INSERT into the OLAP store
t=+5min:  4 minutes of clicks replayed → counted twice
t=+1day:  Invoices go out. 3 large advertisers overbilled ~1.8%
t=+9day:  Advertiser's third-party tracker flags the discrepancy
t=+30day: Credits issued, finance restates revenue, trust damaged

With a correctness design:
t=0:      Same OOM, same restart
t=+40s:   Replay from checkpoint — sink is an idempotent upsert
          keyed by (ad_id, window_start, job_epoch)
t=+40s:   Replayed windows overwrite themselves. Zero double count.
t=+1day:  Nightly batch recount agrees with streaming within 0.02%
t=+1day:  Invoices generated from the reconciled ledger, not the stream

Executive Summary

If you only read one section, read this. The whole case study is one idea: the dashboard number and the invoice number are different products with different correctness bars, and the design must make that explicit.

What This Interview Actually Tests#

Ad click aggregation is not a streaming question. Everyone can draw Kafka → Flink → Druid.

This is a financial correctness under time pressure question that tests:

• Whether you separate billing-grade counts from real-time counts before drawing anything
• Whether you can define "exactly-once" end to end — source, processing, sink — and say where it breaks
• Whether you know that late, duplicate and fraudulent clicks are the normal case, not the edge case
• Whether you have a correction story: when the number was wrong yesterday, how does it get fixed, and who is told?

The key insight: every click is money, but not every click count needs to be money-grade. Staff engineers build two products on one log — a fast approximate one and a slow audited one — and own the reconciliation between them.

The L5 → L6 → L7 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll anchor on billing-grade counting, because that's where the money and the trust live. The real-time dashboard and the pacer hang off the same raw log with weaker guarantees. If I design for the dashboard first, I'll end up with a fast number that finance can't sign."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Ad counting is one of the few domains where the big players published their designs. Citing them shows you know the problem is about correctness, not plumbing.

Google — Photon (joining queries and clicks exactly once)#

Google's Photon paper (SIGMOD 2013) describes joining the ad query log with the click log in real time across multiple datacenters with exactly-once semantics. The key mechanism is the IdRegistry: a Paxos-replicated store of event IDs that have already been joined, so a click processed in two datacenters is only emitted once. Photon tolerates datacenter loss while keeping the output at-most-once and eventually-exactly-once.

Staff insight: Exactly-once at Google's scale was solved with a global dedup registry keyed by event ID, not by trusting the stream engine. In an interview, the speakable version is: "dedup is a first-class component with its own SLO, not a side effect of checkpointing."

Google — Mesa (geo-replicated ads reporting warehouse)#

Mesa (VLDB 2014) is the store behind Google's ads reporting. It ingests updates in versioned batches every few minutes, keeps pre-aggregated tables keyed by dimensions, and makes each version atomically visible across datacenters. Corrections are just new versions; queries read a consistent version.

Staff insight: Versioned, atomically-published aggregates are the clean answer to "how do corrections and backfills not corrupt live dashboards?" You never mutate the number in place; you publish a new version.

Uber — Exactly-once ad events with Flink, Kafka and Pinot#

Uber Engineering described its ad event pipeline for Uber Eats: Flink jobs with exactly-once checkpoints, Kafka transactions between stages, deduplication on a record UUID, and Apache Pinot as the serving layer with upsert support so replays overwrite rather than add. A separate reconciliation path compares against offline Hive data.

Staff insight: This is the modern kappa-plus-reconciliation shape. Even with end-to-end exactly-once in the stream, they still kept an offline comparison — because the stream proving itself correct is not an audit.

Metamarkets → Apache Druid#

Druid was built at Metamarkets specifically for interactive analytics over programmatic ad-exchange data: roll-up at ingestion, columnar segments, bitmap indexes, and sub-second slice-and-dice over billions of events. It is now a common choice for real-time ad dashboards.

Staff insight: The OLAP store choice is driven by the query shape (group-by over a few dimensions with time filters), not by the write rate. Roll-up at ingest is the cost lever: 100× fewer rows if dimensions are bounded.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: The redirect edge mints the click_id and never waits on Kafka — a local spool absorbs Kafka hiccups. Kafka plus the raw archive is the system of record. The streaming path serves dashboards and pacing in seconds; the batch path recounts from raw and writes the ledger that invoices read. The reconciler is the component most designs forget: it compares the two and publishes corrections as new versions instead of mutating in place.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 20 minutes drawing the click → Kafka → Flink → database pipeline, then discover at minute 35 that the interviewer wanted to know what happens to yesterday's invoice when a fraud ring is found. The phases below compress the pipeline to under 10 minutes so the rest goes to correctness, lateness, fraud and corrections — the things that actually level you.

Phase 1: Requirements & Framing (2–3 min)#

State functional requirements in 30 seconds:

"Users click ads, we redirect them to the advertiser, and we aggregate clicks by ad, campaign and time so advertisers can see performance and we can bill them."

Then spend the remaining time on the non-functional split, which is the whole interview:

"This count has three consumers with incompatible needs. Billing needs it exact and auditable, and can wait a day. Advertiser dashboards need it within a minute and can be off by a percent. Budget pacing needs it within seconds and should over-count rather than under-count. I'll anchor on billing-grade correctness and hang the real-time paths off the same log."

Then commit to scale and constraints:

"I'll assume 1B clicks a day — about 12K/s average, 50K/s peak — 10M active ads, redirect p99 under 50ms, dashboards fresh within a minute, and billing final at T+3 days. I'll treat duplicates, late events and invalid traffic as the normal case."

🎯 Staff Move: Saying "billing closes in batch, dashboards close in the stream" in the first three minutes tells the interviewer you know where the bodies are buried. It also licenses you to make the streaming path simpler later.

Phase 2: Core Entities & API (1–2 min)#

Name the nouns, not the schema:

• ClickEvent: click_id (server-minted, signed), impression_id, ad_id, campaign_id, advertiser_id, user_key (hashed), ts_event, ts_ingest, ip, ua, geo, ivt_score
• Aggregate: (ad_id, window_start, dims…) → clicks, valid_clicks, spend_micros, version
• LedgerEntry: (advertiser_id, day, campaign_id) → billable_clicks, amount, version, closed_at

Click path (hot, user-facing):

GET /c/{signed_token}
  → 302 Location: https://advertiser.example/landing?...&gclid-like param
  (token encodes impression_id, ad_id, landing URL, expiry, HMAC)

Query path (advertiser-facing):

GET /v1/reports?advertiser=A&campaign=C&from=...&to=...&granularity=hour&dims=geo,device
  → { rows: [...], data_freshness_ts, is_final: false }

🎯 Staff Move: "Every report response carries data_freshness_ts and is_final. That one field is the contract that lets the dashboard be fast and the invoice be right without anyone calling support about the difference."

Phase 3: High-Level Architecture (≤5 min)#

Staff candidates spend under 5 minutes here. Draw the two paths and say one sentence per box.

"The redirect edge is the only synchronous component. Everything after Kafka is asynchronous. The raw log is the source of truth; the OLAP store and the ledger are both derived from it, at different speeds and with different guarantees."

Phase 4: Transition to Depth#

The sentence that steers the interviewer:

"The pipeline is the easy part. The three places this design actually breaks are: double-counting on replay, late and offline clicks crossing the billing boundary, and fraud discovered after the invoice goes out. I'd like to go deep on exactly-once end to end first, because that's where the money is — then late data, then corrections."

Phase 5: Deep Dives (25–30 min)#

Pick 3–4 based on interviewer signals. Have all of these ready:

Phase 6: Wrap-Up (2–3 min)#

Close with constraints, evolution and what you'd build later:

"To summarize: one immutable click log, two derived products. Streaming gives dashboards and pacing in seconds with a 0.5% accuracy budget. Batch recount gives the ledger finance signs, with a 72-hour lateness window and versioned corrections. The things I'd build next are impression-click joins for CTR and attribution, a self-serve backfill tool with approval gates, and a per-advertiser discrepancy report so sales sees problems before the advertiser does. What I wouldn't build on day one is multi-region active-active counting — regional pipelines that merge at the ledger are enough until a region-loss SLA demands more."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Ad click aggregation is a counting problem where the count is revenue. That one fact turns a textbook streaming exercise into a test of correctness discipline, operational ownership and organizational judgment. At a large ad network, a 1% overcount on $10B of annual click revenue is $100M of refunds and a credibility problem with every agency that runs a third-party tracker against you. A 1% undercount is the same money left on the table. And the latency-sensitive consumers — pacing and fraud — cannot wait for the exact answer.

Interviewers pick it because the L5 answer (a correct streaming pipeline) is plausible and complete-looking, and the L6 answer differs almost entirely in judgment: which number is final, when, and who signs it.

1.2 The L5 → L6 → L7 Contrast — Visual#

The L5 path is not wrong — every box is necessary. It's that the L5 path produces one number with no stated correctness envelope, and the first incident (a restart, a fraud ring, an offline-click backlog) has no designed answer.

1.3 The Staff Question That Cuts Through Everything#

"When this number is wrong, who finds out first — us, or the advertiser?"

If the answer is "the advertiser", there is no reconciliation. If the answer is "us, within a day, with a diff report and a correction already queued", the design is Staff-grade. Every component in this case study exists to make that answer true.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Billing-grade counting. The output is a ledger: billable clicks and amounts per advertiser per day, with a version number and a closed flag. It must be reproducible from the raw click log, defensible in a dispute, and stable once closed. Latency is measured in hours to days. The correctness bar is roughly ≤0.01% disagreement with an independent recount, and every billed click must be traceable to a click_id. Failure is financial: refunds, revenue restatement, and lost advertiser trust. The owner is usually a billing or ads-finance engineering team with finance as the business signatory.

Real-time advertiser dashboards. The output is a set of rollups queried by advertisers and account managers: clicks, spend, CTR by campaign/ad/geo/device/hour. Freshness under one minute, p95 query latency under 1–2s, and an accuracy budget of ±0.5–1% while not final. Failure is reputational: an advertiser sees a number on the dashboard that does not match the invoice and opens a ticket. The owner is the ads reporting team.

Budget pacing & fraud response. The output is a control signal: "campaign C has spent 98% of its daily budget — stop serving" or "this publisher's click rate jumped 40× — quarantine." Freshness in seconds. Accuracy is asymmetric: over-counting causes slightly early budget stops (some lost revenue), under-counting causes overspend that the network usually eats because advertisers are not charged beyond budget. The owner is ads delivery / ads quality.

🎯 Staff Move: "Pacing and billing look like the same counter, but they fail in opposite directions. The pacer should over-count — stopping early costs us a little revenue. The ledger must not over-count — over-billing costs us trust and refunds. That's why they can't share a counter."

2.2 When NOT to Use a Real-Time Click Aggregator#

• Small volume, monthly invoicing. Under ~1M clicks/day with monthly billing and no live dashboard requirement, a nightly SQL job over a warehouse table is correct, cheap and operable by one engineer. Streaming adds an on-call rotation for no product benefit.
• Dashboards that nobody watches live. If advertisers check reports once a day, hourly micro-batch (Spark every 15 minutes) delivers 95% of the value at a fraction of the operational cost.
• Conversion-based billing (CPA). If you bill on conversions reported by the advertiser days later, the click count is an input to attribution, not the billing number. Design the attribution join first.
• Impression-level analytics with unbounded dimensions. If product wants arbitrary slicing over raw events, you need a lakehouse with a query engine, not a pre-aggregating stream.

🎯 Staff Move: "If billing is monthly and nobody needs sub-hour freshness, I'd ship a batch pipeline and revisit when pacing becomes a requirement. Streaming is a cost we take on for a consumer, not a default."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

Each fault line is a place where two competent engineers will disagree. The Staff move is to name the tension, pick a default, and say who pays.

3.1 Fault Line 1: Freshness vs Correctness#

The tension: A number shown at T+30s will be wrong — late clicks haven't arrived, fraud hasn't been scored, duplicates from slow retries haven't been seen. A number shown only when final is useless for pacing and frustrating for advertisers.

Staff default: Two closes. The stream closes a minute window when the watermark passes window_end + 30s and marks it provisional. The batch close happens at T+3d and marks the day final. Any late click between the two lands in a correction stream that updates the provisional number in place (by version).

Watermark math, made concrete: if 97% of clicks arrive within 5s, 99% within 30s, and 99.9% within 10 min, a 30s watermark delay means dashboard minute windows are ~99% complete when first shown and the remaining ~1% arrives as corrections. Billing at T+72h captures >99.99%.

When to deviate: If the product has no live dashboards (monthly reporting), drop the stream close entirely. If pacing is the main consumer (e.g., tight daily budgets on small advertisers), make the pacer even fresher — processing-time counters with no watermark at all — and accept that it over-counts.

🎯 Staff Move: "I'll show two numbers to advertisers: today's provisional spend, updated every minute, and yesterday's final spend once billing closes. The UI labels which is which. That label saves more support tickets than any accuracy improvement."

3.2 Fault Line 2: Exactly-once vs Throughput & Cost#

The tension: End-to-end exactly-once requires dedup state, checkpointed operator state, and a sink that is either transactional or idempotent. Each adds latency, memory and failure surface. At-least-once is simpler and faster but double-counts on every restart and retry.

Staff default: Dedup by server-minted click_id in keyed state with a 24h TTL. Aggregate by (ad_id, window_start). Write the full window value with an upsert keyed by (ad_id, window_start, dims) — never count = count + n. Kafka transactions only between internal stages where downstream is also a stream.

Dedup sizing: 1B clicks/day × 24h TTL × ~16 bytes key ≈ 16 GB raw; with RocksDB overhead ~30–50 GB spread across, say, 64 parallel tasks ≈ under 1 GB per task. A Bloom filter front can cut RocksDB lookups by ~95% for the non-duplicate majority.

Where exactly-once stops: the redirect edge. If the redirect server returns the 302 and crashes before the event reaches Kafka or the local spool, the click is lost. That is at-most-once at the edge — and it's the only place the system under-counts. Fix: write to local disk spool synchronously (≈0.1ms on NVMe with group commit) before the 302, then ship asynchronously.

When to deviate: For derived metrics nobody bills on (CTR heatmaps, creative experiments), at-least-once with HyperLogLog or approximate counts is fine — the exactly-once tax isn't worth it.

3.3 Fault Line 3: Lambda vs Kappa#

The tension: Lambda runs a streaming path and a batch path, merging at serving. It gives an independent truth but two implementations of the same business logic that inevitably drift. Kappa runs one streaming implementation and reprocesses from the log for corrections — one codebase, but no independent check, and replay is bounded by log retention.

Staff default: Write dedup, IVT rules and attribution once as a library (e.g., a pure function over a click record plus lookups). Run it in the stream job and in the nightly batch job over the raw archive. The batch output is the ledger; the diff between batch and stream is a monitored metric (recon.drift_pct), and corrections are published to the OLAP store as a new version for the affected windows.

When to deviate: If you genuinely cannot share code between stream and batch (different languages, legacy Hive SQL), accept lambda but make the diff metric and an owner for it non-negotiable. If billing is low-stakes (internal chargeback), pure kappa is fine.

🎯 Staff Move: "The question isn't lambda versus kappa, it's whether you have an independent recount. I'll run the same logic library in both, and treat batch as the auditor, not as a second product."

3.4 Fault Line 4: Fraud Filtering Inline vs Post-hoc#

The tension: Inline filtering blocks invalid clicks before they're counted — fast and clean, but limited to cheap signals and false positives can't be undone easily. Post-hoc classification uses richer signals (cross-session patterns, publisher-level anomalies, ML models) but arrives hours or days later, after dashboards and possibly invoices have shown the number.

Staff default: Inline: signature validation on the click token, known-bot user agents, datacenter IP ranges, per-user_key and per-IP rate caps (e.g., >5 clicks on the same ad in 60s). Post-hoc within 72h: ML model over session and publisher features. Every click carries an ivt_version; aggregates store clicks_total and clicks_valid; billing reads clicks_valid at close.

Who pays when fraud is found after close: the policy question. Staff answer: credits on the next invoice, issued automatically below a threshold (e.g., <2% of the advertiser's monthly spend), with account-manager approval above it.

When to deviate: In-house ads with no external advertisers (e.g., internal cross-promotion) can skip post-hoc entirely.

3.5 Fault Line 5: Pre-aggregation vs Query Flexibility#

The tension: Rolling up at ingest (minute × ad × geo × device) cuts storage and query cost by 50–500×, but every dimension not in the rollup is unanswerable without going back to raw. Keeping raw events in the OLAP store answers anything but costs 10–100× more and queries slow down as data grows.

Staff default: Druid or Pinot with ingest-time rollup on bounded-cardinality dimensions (country, device type, placement, ad, campaign). High-cardinality dimensions (zip code, user, URL) only in the lake. Every new dimension request is priced: cardinality × time buckets × retention.

Cardinality math: 10M ads × 1,440 minutes = 14.4B potential rows/day — but only active (ad, minute) pairs exist, typically ~1–5% → ~150–700M rows/day before other dims. Adding country (×~20 effective) and device (×3) multiplies the active combinations by maybe 5–10×, not 60×. Adding zip code (×40K) makes the rollup close to raw — that's the line.

🎯 Staff Move: "I'll roll up on dimensions with cardinality under a few hundred. Anything finer goes to the lake. When product asks for zip code on the dashboard, the answer is a price, not a no."

4. Failure Modes & Operational Reality#

4.1 Replay Double-Count — The Restart That Overbills#

Scenario: A TaskManager OOMs during a traffic spike. The job restarts from a checkpoint taken 4 minutes earlier. The sink is an append-style insert.

t=0:       TaskManager OOM (state backend memory pressure)
t=+30s:    JobManager restarts job from checkpoint #18422 (t=-4min)
t=+40s:    Source rewinds Kafka offsets 4 minutes
t=+40s:    4 min of windows re-emitted → appended to OLAP
t=+5min:   Dashboard shows a 2x spike for 4 minutes across all ads
t=+20min:  Nobody notices — spike looks like traffic
t=+1day:   If billing reads the stream: overbill ~0.3% of daily volume

Detection: sink.rows_written vs source.records_consumed ratio jumps; recon.drift_pct exceeds 0.5% at the next reconciliation; job.restarts_total correlated with spike.

Mitigation: Make the sink idempotent: upsert keyed by (ad_id, window_start, dims) with the full window value. Druid/Pinot: use upsert tables or replace segments by interval. Never INCR.

Prevention: Contract test in CI: kill the job mid-window, restart, assert sink totals equal a clean run.

Owner: Reporting pipeline team. Finance is notified if the reconciler shows drift >0.1% on any billed day.

4.2 Stalled Watermark — Windows Never Close#

Scenario: One Kafka partition stops receiving events (a producer region drained for maintenance). The watermark is the minimum across partitions, so it stops advancing. No windows fire. Dashboards freeze; the pacer is fed nothing.

t=0:       Region B redirect fleet drained; its partitions go idle
t=+1min:   Global watermark stuck at t=0
t=+5min:   Dashboards show flat zero for the last 5 min
t=+12min:  Pacer sees no spend; campaigns overspend budgets
t=+20min:  Advertiser support: "my campaign stopped reporting"

Detection: watermark.lag_seconds > 120s; window.fired_total rate drops to zero; per-partition last_event_age_seconds.

Mitigation: Idle-source detection (mark partitions idle after 30–60s so they don't hold back the watermark). Pacer falls back to processing-time counters when event-time lag exceeds 60s.

Owner: Stream platform team owns watermark config; ads delivery owns the pacer fallback. Details of watermark mechanics: Stream Processing.

4.3 Hot Ad — One Campaign at 40% of Clicks#

Scenario: A Super Bowl spot drives a single ad to 40% of all clicks for 10 minutes — ~20K clicks/s on one key.

t=0:       Ad airs; clicks on ad X go from 50/s to 20,000/s
t=+10s:    The task owning ad X hits 100% CPU
t=+30s:    Backpressure propagates to the source; all ads lag
t=+2min:   Checkpoints time out (barrier stuck behind hot task)
t=+5min:   Job restarts — and replays straight back into the hot spot

Detection: task.busy_ratio skew (one task at 100%, median at 30%); checkpoint.duration > timeout; consumer.lag rising on all partitions.

Mitigation: Two-stage aggregation. Stage 1 keys by (ad_id, hash(click_id) % 16) and pre-aggregates per minute; stage 2 keys by ad_id and sums 16 partial counts. Dedup happens before stage 1 keyed by click_id, which is uniformly distributed. Counts remain exact; hot-key throughput scales 16×.

Prevention: Always run two-stage for the click aggregation (the cost is ~10% more CPU and one extra shuffle). Pre-announced big events get a capacity reservation.

Owner: Reporting pipeline team; ads sales gives 48h notice for tentpole events.

4.4 Lost Clicks at the Edge — The Silent Undercount#

Scenario: Kafka brokers in one AZ degrade; producer buffers fill; the redirect service is configured to drop events when its in-memory buffer is full (to keep redirect latency low).

t=0:       Kafka produce p99 goes from 5ms to 2s
t=+20s:    Producer buffer (64 MB) full on 30% of redirect pods
t=+20s:    Pods drop events silently, still return 302s
t=+45min:  Kafka recovers
Impact:    ~3% of 45 min of clicks never logged — never billed

Detection: redirect.events_dropped_total (must exist and must page); compare redirect.302_total vs kafka.clicks_produced_total per minute — a gap >0.1% is an alert.

Mitigation: Local disk spool: append to a local log before returning 302 (≈0.1ms with group commit), ship asynchronously, drain on recovery. Size for ≥1h of the pod's traffic.

Owner: Click edge team. This is under-billing — revenue loss, not advertiser harm — which is exactly why it goes unnoticed without an explicit metric.

4.5 Late-Data Surge — Offline Mobile Clicks#

Scenario: A mobile SDK release batches clicks while offline. After a carrier outage, 2M clicks from 6 hours earlier arrive in 10 minutes.

Detection: events.late_total by lateness bucket (>5 min, >1h, >24h); side_output.late_records rate.

Mitigation: Late events beyond allowed lateness go to a side output that feeds a correction job, updating the affected provisional windows as new versions. Billing is unaffected — the batch close at T+72h includes them.

Owner: Reporting team for corrections; mobile SDK team for the batching behavior (which should cap offline buffering at 24h and stamp ts_event from a server-synced clock).

4.6 Reconciliation Drift — Stream and Batch Disagree#

Scenario: An IVT rule change ships to the stream job but not to the batch job (or vice versa). Nightly recon shows a 0.4% gap that grows each day.

Detection: recon.drift_pct per advertiser per day, alert at >0.1% for billed days, >0.5% for any day. logic_library.version mismatch between stream and batch.

Mitigation: Shared logic library with a version stamped on every output row; deployment pipeline refuses to ship a stream version that the batch job doesn't also have.

Owner: Platform team owns the shared library; reporting team owns the recon alert.

4.7 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Deep Flink internals ≠ correct counting. Explaining checkpoint barrier alignment fluently is not the same as designing an idempotent sink.
• Naming lambda architecture ≠ having a reconciliation design. Two paths without a diff metric and an owner is just two sources of disagreement.
• "We'll use Kafka transactions" ≠ exactly-once to the OLAP store. Transactions end at Kafka; the OLAP sink still needs idempotency.
• Knowing Druid features ≠ knowing the query shape. The OLAP choice should follow from dimensions and freshness, not feature lists.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Load balancer and redirect fleet internals beyond "stateless, p99 < 50ms"
• Stream engine internals (barriers, RocksDB compaction) — reference Flink
• Ad serving and auction design — a different system
• Detailed ML fraud models — name the features and the latency, not the model architecture

6.4 Follow-Up Questions to Expect#

1. "How long do you keep click IDs for dedup, and why that long?"
2. "Your job restarted. Walk me through why the counts are still correct."
3. "An advertiser's third-party tracker shows 8% fewer clicks than your invoice. What do you do?"
4. "How do you backfill three weeks of data after a bug in the IVT rule?"
5. "One ad has 40% of traffic. What breaks and how do you fix it without wrong counts?"
6. "Which OLAP store, and what does a typical dashboard query cost?"
7. "When is a day's number final, and who decides?"

7. Active Drills#

Drill 1: The Opening (Intent + Constraints)#

Prompt: "Design a system that aggregates ad clicks in real time."

Drill 2: Exactly-Once, End to End#

Prompt: "You said exactly-once. Prove it — walk me from the click to the dashboard."

Drill 3: Late and Offline Clicks — Make It Concrete#

Prompt: "A click happens at 23:59:50 on Monday on a phone in airplane mode. It arrives at 06:00 Tuesday. Which day is it billed on, and what does the dashboard show?"

Drill 4: The Pipeline Is Down#

Prompt: "Kafka is unavailable for 20 minutes. What happens to clicks, redirects, dashboards and budgets?"

Drill 5: Hot Ad#

Prompt: "A Super Bowl ad gets 40% of all clicks for 10 minutes. What breaks, and how do you keep counts exact?"

Drill 6: Fraud Found After the Invoice#

Prompt: "Three days after billing closed, ads quality identifies a click farm that generated 4% of one advertiser's clicks last week. What happens?"

Drill 7: Build vs Buy for the OLAP Store#

Prompt: "Druid, Pinot, ClickHouse, BigQuery, or build on Cassandra? Pick one and defend it."

Drill 8: Changing an IVT Rule Without an Incident#

Prompt: "Ads quality wants to tighten the click-rate rule from 5 to 3 clicks per user per ad per minute. How do you ship it?"

Drill 9: Cost — Cut the Bill by 40%#

Prompt: "The aggregation platform costs $250K/month. Finance wants 40% off. Where do you cut?"

Drill 10: Multi-Region#

Prompt: "We're launching in the EU with data residency. How does counting work across regions?"

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Incident — The Super Bowl Spike#

Context: During a major televised event, click volume goes from 15K/s to 90K/s in under a minute. Consumer lag climbs to 8 minutes, dashboards freeze, and the budget pacer is starving. Three large advertisers' campaigns have overspent their daily budgets by 30%. The on-call escalates to you.

Questions to Surface First:

• Is the lag uniform across partitions, or concentrated on tasks owning a few hot ads?
• Is the pacer reading event-time windows (now 8 minutes stale) or processing-time counters?
• Are clicks being lost anywhere, or only delayed? Check redirect.302_total vs kafka.produced_total.
• Who decided the overspend policy — does the network absorb it, or is it billed?

Typical L5 Approach: Scales the stream job (more TaskManagers, more parallelism), waits for lag to drain, and confirms no data loss. Correct, but treats it as a throughput problem. If the hot key is the issue, more parallelism doesn't help; and the overspend continues while it drains.

Staff Approach: Separates the two problems: pacing is a control loop that must not depend on a lagging pipeline, so switch the pacer to its processing-time fallback immediately and apply conservative throttling to campaigns near budget. Then diagnose the pipeline: busy-ratio skew shows two hot ads; confirm two-stage aggregation is enabled (it was disabled in a config refactor). Re-enable, restart from checkpoint, lag drains in ~6 minutes. Overspend is credited per policy.

Principal Approach: Treats the incident as evidence that tentpole events have no owner. Creates a "tentpole readiness" process with sales: events are registered 48h ahead with expected volume, the pipeline is pre-scaled, a named incident commander is on call, and pacing switches to conservative mode during the event window by default. Prices the overspend: credits issued vs cost of pre-scaling (a few thousand dollars of compute) — the readiness process pays for itself in one event.

Key Takeaway: "Pacing is a control loop. Never let a control loop depend on a pipeline whose latency you don't control."

What clears the Staff bar:

• Splits the control-loop problem from the throughput problem
• Diagnoses hot key vs capacity with a specific metric
• Ties overspend to a named cost owner

Deep Dive 2: Silent Failure — The 3% Undercount Nobody Noticed#

Context: Finance notices that click revenue for the last six weeks is ~3% below forecast with no traffic change. Advertisers aren't complaining — they're being under-billed. You're asked to find out why.

Questions to Surface First:

• Do edge counts (redirect.302_total) match logged clicks (kafka.produced_total)?
• Did dedup start dropping legitimate clicks — e.g., a click_id collision?
• Did an IVT rule change ship around six weeks ago?
• Does the batch recount agree with the stream? (If both agree, the loss is upstream of both.)

Typical L5 Approach: Checks the stream job for errors, finds none, checks the reconciliation, finds stream and batch agree. Concludes the counts are correct and traffic must have changed.

Staff Approach: Stream and batch agreeing means the loss is upstream of the log. Compares 302 count vs produced count per pod: a new redirect release six weeks ago changed the Kafka producer buffer config and drops events on buffer-full under load — about 3% at peak. No metric existed for dropped events. Fix: local disk spool and a redirect.events_dropped_total alert. Revenue is unrecoverable for clicks never logged — except that the 302 access logs on the load balancer still have them for 30 days; a one-time recovery job reconstructs the last 30 days from those logs.

Principal Approach: Recognizes that reconciliation only compares two things derived from the same log — it cannot detect losses before the log. Mandates an edge-to-ledger completeness check as a standard for every revenue pipeline: an independent count at the earliest point (load balancer access logs) reconciled daily to the ledger. Adds "revenue completeness" to the monthly business review so a 3% gap surfaces in days, not weeks.

Key Takeaway: "Reconciliation between two derived counts can't detect loss upstream of both. You need one count taken before the log."

What clears the Staff bar:

• Reasons about what reconciliation can and cannot detect
• Finds an independent source for recovery
• Turns a missing metric into an ownership fix

Deep Dive 3: Large Customer Onboarding — An Agency Brings 20% of Volume#

Context: Sales signs a holding company whose agencies will run ~20% of total click volume, starting in 3 weeks. They require hourly reports within 15 minutes, a contractual discrepancy cap of 3% vs their third-party tracker, and a monthly reconciliation file at click level.

Questions to Surface First:

• What does their third-party tracker count — clicks or landing-page arrivals? (The definitions differ by 5–10% naturally.)
• Do they need click-level data exports? What PII is in a click record?
• Is 20% of volume concentrated in a few campaigns (hot keys) or spread?
• Who agreed to 3%, and is it measurable with our definitions?

Typical L5 Approach: Capacity-plans for +20% throughput, adds an export job for click-level files, and adds a dashboard for the agency. Treats it as scaling work.

Staff Approach: The capacity is the easy part. The 3% cap is the risk: our click count and their tracker's landing-page count differ structurally (bounces before the landing page loads, IVT we filter that they don't). Builds a per-advertiser discrepancy report that compares our counts with theirs daily and alerts at 2%, so account managers see problems before the agency. The click-level export is hashed IDs only, through the existing data-sharing pipeline with legal sign-off.

Principal Approach: Pushes back on the contract term before it's signed: a 3% cap on a metric with a natural 5–10% definitional gap is a commitment engineering can't keep. Works with sales and legal to restate the term in terms of a shared, defined measurement (e.g., clicks with a matching landing-page beacon, using a shared click ID passed as a URL parameter). Turns the bespoke work into a standard "enterprise reporting tier" product with a price.

Key Takeaway: "Discrepancy caps are measurement contracts. Define the measurement before you sign the cap."

What clears the Staff bar:

• Identifies the definitional gap as the real risk, not throughput
• Builds detection that reaches the account team first
• Connects engineering to the sales process

Deep Dive 4: Post-Mortem — Three Weeks of Wrong Invoices#

Context: A change to the attribution logic in the shared library shipped with a bug that counted clicks on video companion ads twice (once for the video, once for the companion). It went unnoticed for three weeks because both stream and batch used the same library. 140 advertisers were overbilled by an average of 1.2%. You're running the post-mortem.

Questions to Surface First:

• Why did reconciliation not catch it? (Same library in both paths — they agreed with each other.)
• Was there any independent signal? (Advertiser discrepancy reports? Edge counts?)
• How far back can we recompute? (Raw archive retention: 13 months — yes.)
• Which invoices are affected, and what's the credit process?

Typical L5 Approach: Fixes the bug, backfills three weeks, issues credits, adds a unit test.

Staff Approach: Fixes and backfills, but the core finding is that shared logic makes stream and batch agree on bugs. Adds two independent checks: (1) an invariant — billable clicks ≤ 302 redirects at the edge per ad per hour (a click can't be counted more times than it was redirected); (2) a canary comparison — new library versions run in shadow against the previous version for 48h, and any per-advertiser delta >0.2% blocks promotion.

Principal Approach: Recognizes the trade: a shared library eliminated drift but created correlated failure. Establishes that revenue-affecting logic changes follow a financial change process — shadow, delta report, finance sign-off above a threshold — and that the edge-count invariant is a standard for every revenue counter in the company. Owns the external communication with finance: proactive credits and a note to affected advertisers before they find it, which costs less trust than being caught.

Key Takeaway: "Shared logic removes drift and adds correlated failure. Pair it with an invariant that doesn't use the shared logic."

What clears the Staff bar:

• Spots that agreement between paths isn't evidence of correctness
• Adds an independent invariant grounded in the edge
• Separates computing a restatement from approving one

Deep Dive 5: Multi-Region Expansion — Active-Active Redirects#

Context: Leadership wants the click edge in four regions, active-active, so that losing one region doesn't stop redirects. Today everything runs in one US region. Billing must remain correct during and after a regional failover.

Questions to Surface First:

• Does a region failover cause clients to retry the same click in another region? (Yes, if the client retries a failed redirect.)
• Does any data-residency requirement prevent centralizing raw clicks?
• What's the RPO for click logs if a region is lost — can we lose the last N seconds?
• Is the dashboard expected to be globally consistent?

Typical L5 Approach: Replicate Kafka across regions with MirrorMaker, run a global Flink job reading the merged topic, dedup globally.

Staff Approach: Regional cells. Each region mints click_ids with a region prefix and runs its own edge, Kafka, stream and OLAP. Cross-region duplicates only occur on client retry after failover; the signed click token carries the original region, and a small globally-replicated deny-set (24h TTL, ~MBs) rejects the second mint. The ledger merges regional daily aggregates. RPO: the local spool is replicated to a second AZ, so a single-AZ loss loses nothing; a full region loss loses at most the unshipped spool (seconds).

Principal Approach: Asks what failure leadership is actually buying insurance against and prices it: four-region active-active is roughly 3–4× the edge cost and 2× the pipeline cost. If the real need is "redirects keep working", only the edge must be multi-region — the pipeline can stay in two regions with async shipping. Writes down the regional failure posture: which region's billing close can be delayed, who decides, and how advertisers are told.

Key Takeaway: "Mint in one region, dedup in that region. Global coordination only for the rare cross-region retry."

What clears the Staff bar:

• Avoids global dedup by construction
• Quantifies RPO and ties it to the spool design
• Distinguishes edge availability from pipeline availability

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Split "the click count" into billing-grade, dashboard and pacing products, each with a numeric correctness and freshness envelope
• Walk exactly-once end to end — edge spool, idempotent producer, keyed dedup, checkpointed windows, idempotent upsert sink — and name where it breaks
• Set watermark and allowed-lateness per consumer, and explain why billing closes in batch
• Design two-stage salted aggregation that keeps hot-ad counts exact
• Describe kappa-plus-reconciliation and why shared logic needs an independent invariant
• Design fraud reclassification as versioned corrections with a credit policy
• Choose an OLAP store from the query shape and price rollup dimensions
• Run a backfill or restatement with approval gates

The Bar for This Question#

Mid-level (L4): Builds a working pipeline: click service → queue → aggregator → database → dashboard. Knows to use a message queue and windowed aggregation. Correctness under restart, duplicates and late data is not addressed until prompted.

Senior (L5): Builds a correct-looking streaming pipeline with Kafka, Flink, event-time windows and an OLAP store. Mentions dedup and exactly-once. Tends to treat the stream as the billing number, uses one lateness setting, and has no correction or reconciliation story. Fraud is a filter.

Staff+ (L6): Starts from the consumers and their correctness bars. Treats exactly-once as a path property with an honest edge boundary. Separates the provisional stream from the batch-closed ledger, designs reconciliation with drift metrics, versions corrections and fraud reclassifications, and names owners and approval gates for anything that moves revenue. Knows the hot-ad cascade and the stalled-watermark failure. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "The Stream Should Never Be the Billing Number"#

The Staff position: Streaming previews; batch closes. Even with flawless exactly-once, billing needs a close event, a frozen logic version and an independent recount. Uber's Flink pipeline has end-to-end exactly-once and still compares against offline data.

Why this matters in interviews: Candidates who bill from the stream will be walked through restart, late data and fraud until the design collapses. Saying this early ends that line of questioning.

10.2 "Lambda vs Kappa Is the Wrong Debate"#

The Staff position: One logic library, run in streaming and batch modes, plus an invariant that doesn't depend on that library. The architecture label doesn't matter; the audit does.

Why this matters in interviews: Interviewers who ask "lambda or kappa?" are checking whether you know why lambda existed. Answering "I want an independent recount — here's how I get it without two codebases" is the strongest answer.

10.3 "Exactly-Once Is Cheaper at the Sink Than in the Engine"#

The Staff position: Design the output as an idempotent function of (key, window) and you rarely need transactional sinks. Spend the engineering effort on the key design.

Why this matters in interviews: It shows you understand why exactly-once works, not just that a flag enables it.

10.4 "Most Click-Count Disputes Are Definition Disputes, Not Bugs"#

The Staff position: Build a discrepancy report and a published measurement definition before investing in more accuracy. Most tickets close with an explanation, not a fix.

Why this matters in interviews: It shows you know where the business pain actually comes from.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer designs a click aggregator. A Principal engineer notices that search ads, display ads, video ads and the new retail-media product each count clicks in their own pipeline, each with its own dedup, its own IVT rules and its own definition of a billable click — and that finance reconciles four ledgers by spreadsheet every month. The L7 question is not "Flink or Spark?" but "how many ways does this company count money, who certifies each, and which do we consolidate?" At org scale, click aggregation is a measurement platform with a governance layer: one event schema, one dedup service, one IVT methodology with versioning, one ledger contract with finance, and many surfaces plugged into it.

🧭 Principal Move: "Before designing another pipeline, I'd like to know how many systems compute billable events today and how finance reconciles them. If the answer is 'four, by spreadsheet', the highest-leverage design is a shared measurement platform with one ledger contract — and the new surface becomes its first customer."

The Org-Level Fault Line#

One measurement platform vs per-surface counting pipelines.

The deciding question: how many billable surfaces will exist in 2 years? One or two: shared library and conventions. Three or more: the platform pays for itself in finance hours and avoided restatements.

Cost Model#

Assumptions: cloud list prices, ~300 bytes per raw click, Parquet compression ~5×, managed Kafka, a self-managed or managed OLAP cluster, loaded engineer cost ~$25K/month.

The line that matters to leadership: at 1B clicks/day and a ~$0.50–$1 average CPC, the platform counts ~$15–30B a year. A $150K/month platform is ~0.01% of what it measures. The expensive failures are not infrastructure — a single 1% overbilling event costs more than a year of the platform.

The 3-Year Evolution Path#

Each step is triggered by a business event. Building regional cells before there's a second region or a residency requirement is how a platform team spends a year on something nobody needs yet.

One-Way Doors vs Two-Way Doors#

🧭 Principal Insight: Spend review time proportional to reversal cost. The OLAP debate gets an afternoon. The click_id format, the billable-click definition and raw retention get a design review with finance, legal and ads quality in the room.

The Standard I'd Write#

RFC: Billable Event Measurement Standard (v1)

Scope: Every system that counts events used for invoicing, revenue recognition, or advertiser-facing reporting.

Requirements:

• Billable events MUST carry a server-minted, signed, globally unique event ID assigned at the first server-side touchpoint.
• Raw events MUST be written to the immutable measurement log and retained for ≥13 months.
• Every ledger number MUST be reproducible from the raw log plus a recorded logic version.
• Sinks feeding reports or ledgers MUST be idempotent; append-only increments are not permitted.
• Pipelines MUST publish recon.drift_pct against an independent count (edge or batch) and page at >0.1% on billed days.
• Changes to IVT rules or attribution logic projected to move revenue >0.5% MUST run in shadow ≥7 days and be approved by finance.
• Advertiser-facing reports SHOULD expose is_final and data_freshness_ts.

Exceptions: Filed with the measurement architecture group; decision within 5 business days; maximum duration 2 quarters.

Success metrics: All billable surfaces on the platform within 4 quarters; zero restatements caused by double-counting; finance monthly reconciliation time reduced from days to hours; advertiser discrepancy tickets down 50%.

What I'd Tell the VP#

"We count roughly $20 billion a year of clicks across four separate systems, and finance reconciles them by hand every month. Each system has its own idea of what a valid click is, which is why advertisers sometimes see different numbers for the same campaign. I'm proposing we move all of them onto one measurement platform over the next year, with a fast preview number for dashboards and an audited number for invoices. It costs about six engineers for a year. The return is fewer refunds, a month-end close that takes hours instead of days, and a fraud system that can see across all our ad products. The biggest risk is migration — we'll run old and new side by side and only switch billing when they agree for 30 days."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll reconcile stream against batch nightly." — correct, but silent on who acts on the diff, what threshold triggers a restatement, and who approves it.
• "We'll use two-stage aggregation for hot ads." — right mechanism, but no capacity process for tentpole events that the whole org can plan around.
• "Pinot for serving." — sound, but never asks whether the org already runs an OLAP engine, or what a second one costs to operate for five years.

Appendices

clicks
  .assignTimestamps(e -> e.ts_event, maxOutOfOrder = 30s)
  .withIdleness(60s)                       // idle partitions don't stall watermark
  .keyBy(e -> e.click_id)
  .filter(dedupWithTtl(24h))               // RocksDB seen-set; dups to side output
  .map(applyIvtRules(rulesAt(e.ts_event))) // effective-dated rule version
  .keyBy(e -> (e.ad_id, hash(e.click_id) % 16))
  .window(Tumbling(1 min)).allowedLateness(5 min)
  .aggregate(sumClicksAndSpend)            // stage 1 partials
  .keyBy(p -> p.ad_id)
  .window(Tumbling(1 min))
  .reduce(sumPartials)                     // stage 2 exact totals
  .sinkTo(upsert(key = (ad_id, window_start, dims), value = full_totals))

lateSideOutput -> correctionJob -> upsert(version = n+1)

GET /v1/reports?advertiser=A&from=2026-09-01&to=2026-09-30&granularity=day&dims=campaign
200 OK
{
  "rows": [...],
  "data_freshness_ts": "2026-10-01T09:14:00Z",
  "final_through": "2026-09-27",       // days closed for billing
  "is_final": false,
  "logic_version": "attrib-v42/ivt-v17"
}

# Completeness
redirect.302_total                      # independent edge count
redirect.events_dropped_total           # must be zero; page on any
edge_to_log.gap_pct                     # 302s vs produced, per minute

# Correctness
dedup.duplicates_total                  # expected ~0.5-2%
recon.drift_pct{advertiser,day}         # stream vs batch
invariant.billable_gt_redirects_total   # must be zero

# Freshness
watermark.lag_seconds
olap.ingest_lag_seconds
events.late_total{bucket}

# Fraud
ivt.rate{rule_version,surface}
ivt.reclassified_after_close_total