Technologies referenced in this case study: Redis · Apache Kafka · Apache Flink · Cassandra · DynamoDB · OLAP Databases

Related: Ad Click Aggregator · Rate Limiter · Distributed Cache · Ledger and Digital Wallet · News Feed · Feature Flags · Hot Keys · Graceful Degradation · Latency & Protocols

Reading Guide#

Organized for interview use first, reference second. Read front-to-back once, then return to the fault lines and incidents that match your weak spots.

Without distributed budget control:
t=0:        A $50K daily-budget campaign matches a trending topic. Win rate jumps 30×.
t=+1min:    Ad servers check spend from a counter updated by a batch job every 5 minutes.
            The campaign wins ~180K impressions/min at $9 CPM → $1,620/min.
t=+5min:    Batch job lands: spend $8.1K. Counter still says "under budget" for the next window.
t=+31min:   Spend crosses $50K. Batch lags; servers keep serving for another 5 minutes.
t=+36min:   Campaign stops at $58K. $8K overdelivered. Advertiser contract: pay only up to budget.
            The $8K is revenue the platform gave away — and impressions other advertisers lost.

With leased budget slices and pacing:
t=0:        Same viral match. Each ad server holds a small budget lease (~$5) for this campaign.
t=+1min:    Leases are consumed in seconds; servers ask the budget service for more.
            Pacing controller sees spend running 30× ahead of plan → throttle to 4% participation.
t=+8h:      Campaign spends evenly through the day, ends at $50,040.
            Overdelivery bounded by outstanding leases: ~$40 across 8 regions.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What the Interviewer Is Scoring#

Ad serving is not a ranking question. Everyone can sort by bid × pCTR.

It is a money-under-a-deadline question that tests:

• Whether you split a fixed ~100 ms budget across stages and decide what to drop when a stage is slow
• Whether you know the auction rule is an economic contract — it changes how advertisers bid, not just who wins
• Whether you can bound budget overspend with a number, and say who absorbs it
• Whether you distinguish served, rendered and billable impressions, and dedupe the events that become invoices

The key insight: Every ad decision spends someone's money with incomplete, slightly stale information. A Senior design optimizes the ranking. A Staff design bounds the error: how stale the spend counter may be, how much a campaign may overdeliver, how many duplicate impressions reach billing, and what the system serves when the deadline hits. Revenue is a function of the ranking; trust is a function of the bounds.

One Question, Three Levels#

Positions to Commit To#

Which Problem Are We Solving?#

Three intents produce three different systems. Name them, then commit.

🎯 Staff Move: "I'll design the owned-and-operated case — ads inside our own feed and search, our advertisers, our auction. That's where budget, pacing, auction rules and billing all live in one system. If we were an exchange, the core problem would be fanning out to external bidders under a deadline; if we were a bidder, it would be deciding what not to bid on. I'll mention where those differ."

Where the Design Splits#

How Real Companies Built It#

Why this section belongs here: The auction rule, the request deadline and the budget contract are all things large platforms have documented publicly. Citing them shows you know these are deliberate product decisions, not implementation details.

Google Ad Manager — Moving the Whole Platform to First-Price Auctions#

In September 2019 Google announced it was completing the rollout of a unified first-price auction across Google Ad Manager, after tests that it described as showing a neutral to positive impact on publishers' total revenue and a more competitive market in which third-party buyers won an increased share of impressions (Google Ad Manager blog).

Staff insight: A pricing-rule change at that scale was announced, staged and justified with test results — because it changes every buyer's bidding strategy. In an interview, say: "The pricing rule is a marketplace contract. Changing it is a migration for every advertiser's bidding tools, so it ships with notice and holdouts, not a config flip."

OpenRTB — The Deadline and the Auction Type Are Fields in the Request#

The IAB Tech Lab's OpenRTB 2.6 specification puts both decisions in the bid request itself: tmax is the maximum time in milliseconds the exchange allows for bids to be received, including internet latency, and at is the auction type, where 1 means first price and 2 means "second price plus" (the default). A bidder signals no-bid with an empty HTTP 204 response, and the spec's example request carries "tmax": 120 (OpenRTB 2.6).

Staff insight: The industry standard treats the latency budget and the pricing rule as explicit per-request parameters, not implicit behavior. Do the same internally: every stage receives an absolute deadline, and the auction type is a documented property of the placement.

Google Ads — A Daily Budget That Is an Average, With a Monthly Ceiling#

Google Ads documents that a campaign might spend up to twice its average daily budget on a given day to take advantage of traffic fluctuations, and that over a month it will spend no more than 30.4 times the average daily budget (Google Ads Help).

Staff insight: "Budget" is a contract with a defined tolerance, not an exact per-request limit. That's exactly the framing a Staff candidate should bring: "I'd define the budget promise first — daily, monthly, with what tolerance — and then design enforcement to that bound. An exact per-request global counter is neither necessary nor affordable."

Follow-Ups to Expect#

System Architecture Overview#

Reading the diagram: The gateway stamps an absolute deadline on every request. Retrieval narrows millions of ads to a couple of thousand using an in-memory targeting index and drops campaigns the pacing controller is currently throttling. Ranking scores the survivors in two stages. The auction converts bids and predictions into a winner and a price. The guard stage checks a local budget lease and the frequency counter — no synchronous call to a global budget. Served, rendered and click events flow through a log to a stream aggregator that dedupes by impression ID, filters invalid traffic and feeds both billing and the control loop. The metric that tells you money is safe is campaign.overdelivery_pct; the one that tells you revenue is safe is serve.deadline_miss_rate.

One-Minute Recap#

Numbers to Bring#

Interview Walkthrough

The most common mistake: Candidates spend 15 minutes on targeting criteria, the campaign schema and the ML model architecture, then run out of time before the interviewer asks the questions that decide the level: "A campaign goes viral. How much does it overspend?" and "What do you bill on?" Compress the funnel to ~8 minutes and spend the rest on the latency budget, the auction rule, budget and pacing, frequency caps and the event pipeline.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional scope in one breath:

"When a client renders an ad slot, it asks us for an ad. We pick eligible ads by targeting, predict how likely this user is to click or convert, run an auction to choose a winner and a price, enforce budgets, pacing and frequency caps, return the ad, and record impressions and clicks so advertisers are billed correctly."

Then the non-functional requirements, which is where the design lives:

"Four constraints drive everything. One: a hard deadline — about 100 ms end to end, so roughly 60 ms for our own work, and a late answer is worth nothing. Two: money moves on every decision, so budget overspend must be bounded with a number we can defend. Three: the auction rule is a contract advertisers bid against. Four: billing must be correct and auditable — served is not billed. I'll assume 500K ad requests per second at peak, 5 million active ads from 200K advertisers, 500 million daily users, and a mix of CPM, CPC and CPA bids."

Then name the underspecified parts:

"I'd confirm: is this our own inventory or an exchange? Do we bill on impressions rendered or viewable? What's the budget promise — exact daily, or average with a tolerance? I'll assume owned-and-operated inventory, billing on rendered impressions and clicks, and a daily budget with overdelivery bounded under 1%."

🎯 Staff Move: Saying "a late answer is worth nothing, and overspend needs a number" reframes the problem from "rank ads" to "spend other people's money correctly under a deadline" — that's the sentence that sets the level.

Phase 2: Core Entities & API (1–2 minutes)#

Name the nouns in 30 seconds:

• Campaign: campaign_id, advertiser_id, daily_budget, budget_tz, bid_type (CPM/CPC/CPA), bid_amount, targeting, freq_cap (e.g. 3/day), state
• Ad (creative): ad_id, campaign_id, format, asset_refs, review_state
• Ad request: request_id, placement_id, user_key (or none), context (page, query, geo, device), deadline_ms
• Impression: impression_id (signed), request_id, ad_id, price, auction_type, served_at, rendered_at?
• Click: click_id, impression_id, ts, ip_hash, ua_hash

Serving API:

POST /v1/ads:select
{ placement_id, slots: 1, user_key, context: { geo, device, page, query }, deadline_ms: 80 }
→ 200 { ads: [ { ad_id, creative, price_micros, impression_url, click_url } ] }
→ 204 No ad (nothing eligible or deadline reached with no candidate)

GET  /i?imp=<signed impression_id>          (impression beacon, fires on render)
GET  /c?imp=<signed impression_id>&dest=…   (click redirect, 302 to landing page)

Advertiser API (control plane, not latency-critical):

POST  /v1/campaigns                     { budget, bid, targeting, freq_cap }
PATCH /v1/campaigns/{id}                { budget?, bid?, state? }   → index refresh ≤ 30s
GET   /v1/campaigns/{id}/delivery       spend, impressions, pacing status

🎯 Staff Move: "The impression ID is generated at serve time and signed into the beacon and click URLs. Billing counts each signed ID once. A replayed or forged beacon either fails the signature or dedupes away."

Phase 3: High-Level Architecture (≤5 minutes)#

Draw at most eight boxes:

Walk one request in 90 seconds:

1. The gateway resolves the user and context, stamps deadline = now + 80 ms, and calls retrieval.
2. Retrieval queries an in-memory inverted index over targeting attributes — geo, language, interests, placement — intersecting posting lists to ~2,000 eligible ads, dropping campaigns the pacing controller has throttled out of this request.
3. Ranking fetches features in one batched call, runs a light model on 2,000, then a heavy model on the top 300 to predict pCTR and pCVR.
4. The auction computes eCPM = bid × p(action) × 1000 × quality, applies the placement's reserve price and computes the winner's price under the documented rule.
5. The guard checks the winner's frequency counter for this user and decrements the local budget lease. If either fails, it falls to the next candidate.
6. We return the ad with a signed impression ID. The client fires the beacon on render; clicks pass through the redirect. Events stream to an aggregator that dedupes, filters invalid traffic, bills, and feeds spend back to pacing and budget within seconds.

🎯 Staff Move: Say out loud: "Nothing on the hot path makes a synchronous call to a global counter. Budget is a local lease, caps are a single low-latency read, and the index is in memory. That's how the 60 ms holds at p99." You've now spent ~8 minutes.

Phase 4: Transition to Depth (1 minute)#

"That's the happy path, and it's the Senior-level design. What makes this hard is that we're spending advertisers' money with stale information under a deadline. I'd like to go deep on the latency budget and degradation, the auction and pricing rule, budget enforcement and pacing, frequency caps, and the billing pipeline. Where would you like to start?"

If no preference: start with budget. It's the question with money attached, and it decides the level.

Phase 5: Deep Dives (25–30 minutes)#

For each: state the tradeoff → commit → quantify → name who pays.

Deep dive 1: Budget enforcement and pacing (7–8 min)

"Budget is two problems. Enforcement — don't exceed the budget by more than a bound — and pacing — spend it evenly across the day. For enforcement, the budget service owns remaining spend per campaign and hands each ad server a lease sized to ~10 seconds of that server's expected spend for that campaign. Servers decrement locally. When a lease is 70% used, they ask for more; when remaining budget is under, say, 5%, leases shrink to a few impressions each. For pacing, a controller compares actual spend to a plan curve shaped like the day's traffic and sets a participation probability per campaign every 30 seconds: ahead of plan → lower, behind → higher."

Quantify: "Worst-case overdelivery is outstanding leases plus impressions served but not yet rendered. 400 servers × $0.50 lease ≈ $200 on a $50K campaign — 0.4%. I'd alert per campaign above 1%."

Who pays: "If advertisers are billed only up to budget, overdelivery is free inventory — the platform pays. That's why the bound is a finance number, not just an engineering one."

Deep dive 2: The latency budget and degradation (6–7 min)

"80 ms deadline at the gateway. Retrieval gets 15, features and ranking 25, auction and guard 5, leaving slack for serialization and the network. Each stage receives the absolute deadline, not a relative timeout, so stages can't each assume they have their full share. If the heavy model hasn't returned by its cutoff, the auction runs on light-model scores. If retrieval is slow, we serve from a per-placement cache of recent winners. A late response is a lost impression — so the degradation ladder always ends in a valid ad or a 204, never a timeout."

Quantify: "Fan-out to 50 ranking shards means p99 of the request is dominated by the slowest shard. At 50 shards, a 1-in-100 slow shard hits ~40% of requests — so we hedge or cut off the tail rather than wait."

Deep dive 3: The auction and pricing rule (5–6 min)

"Rank by eCPM × quality, where quality folds in predicted user experience — hide rate, landing-page quality. Pricing on our own inventory: generalized second price — the winner pays the smallest bid that would still have kept its position, plus a cent, never below the reserve. That keeps advertisers from needing to shade. Reserve prices per placement come from experiments, not intuition. The rule and its changes are published; a change ships behind a holdout with advertiser-ROI guardrails."

Deep dive 4: Frequency caps (4–5 min)

"Counters keyed by (user, campaign, day) in a low-latency KV store, ~20 bytes each, TTL 26 hours. One batched read per request for the top candidates' campaigns. Increments happen on render events, not on serve, so the counter can lag by seconds — we accept ±1 impression. If the store is down, we fail open, but with a ceiling: we stop serving campaigns with caps of 1 per day, because those advertisers bought exactly one exposure."

Deep dive 5: Billing pipeline and ownership (3–4 min)

"Billable impression = rendered beacon with a valid signature, first occurrence of that impression ID, not filtered as invalid traffic, within 1 hour of serve. Clicks: first click per impression, fraud-filtered. The aggregator writes billable events to a ledger with the impression ID as the idempotency key, so replaying the event log never double-bills. The ads serving team owns serving latency and pacing quality; the billing team owns the ledger and invoices; the trust team owns invalid-traffic rules; the ML team owns model calibration with alerts on drift."

Phase 6: Wrap-Up (2–3 minutes)#

"The core idea: every ad decision spends someone's money with slightly stale information under a hard deadline, so the design is about bounds. Per-stage deadlines with a degradation ladder; an eCPM auction with a documented pricing rule; budget enforced by leases with a written overdelivery bound and pacing by throttling; approximate frequency caps that fail open with a ceiling; and billing on rendered, deduplicated, filtered events."

The evolution closer:

"What I'd build later: auto-bidding — advertisers set a target cost per action and we bid for them, which makes pacing and bidding one control loop; budget leases across regions; and an advertiser-facing event-level log for audit. What I'd not build: a globally consistent spend counter on the hot path — I'd invest in shrinking feedback lag instead."

🎯 Staff Move: End on who absorbs the error. "Overdelivery is bounded at under 1% and the platform eats it; billing never charges for an unrendered or duplicated impression; and when the deadline hits, the user sees a slightly less relevant ad, not a blank slot."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Ad serving is the interview where every component has a price tag attached to its error. A stale spend counter is overspend. A miscalibrated model is mispriced inventory. A duplicated beacon is an overcharge. A slow feature store is a blank slot. Senior candidates design each component to work; Staff candidates design each component's error to be bounded, measured and owned — and can say who pays when the bound is exceeded.

It also has an unusual feedback structure. The serving path depends on numbers computed by the event pipeline — spend, frequency counts, click rates — and those numbers arrive late. The system is a control loop with lag: serve, observe, adjust. Treating it as a request/response service misses that most incidents are control-loop incidents — pacing that oscillates, budgets that react too slowly, models trained on yesterday's distribution.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"It's 9:05 a.m. A retailer's $50,000 daily campaign just matched a trending topic and its win rate jumped 30×. Walk me through, minute by minute, what your system does with its budget, what the advertiser sees, and how much it overspends by the end of the day."

A candidate who answers with lease drain (seconds), the pacing controller's next tick (30 s) dropping participation, the spend-feedback lag that bounds how far ahead it can get, the overdelivery bound (sum of leases, ~$200), the advertiser's dashboard showing "pacing limited", and the end-of-day result (budget spent evenly, ±0.5%) has run an ad system. A candidate who says "we check the budget before serving" has written a filter.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Owned-and-operated serving → bounded money under a deadline

• Constraint: our inventory, our advertisers, our auction; latency inside our own page render
• Strategy: index retrieval, two-stage ranking, eCPM auction, leases and pacing, caps, billing pipeline
• Failure mode: overspend, miscalibration, deadline misses, overbilling
• Who pays for imperfection: the platform (overdelivery, credits), advertisers (overbilling until disputed), users (bad or repetitive ads), host-page teams (latency)

Ad exchange → fan-out under someone else's deadline

• Constraint: external bidders across the internet; we hold no budgets; tmax is a hard wall
• Strategy: per-bidder timeouts and QPS throttles, floor prices, first-price clearing, creative validation, reproducible auction logs
• Failure mode: slow bidders consume the deadline; bidders sent traffic they never bid on waste everyone's money
• Who pays: publishers (unfilled slots), bidders (wasted compute), the exchange (disputes)

Demand-side bidder → deciding what not to bid on

• Constraint: answer inside tmax minus round-trip network time; most requests should be declined cheaply
• Strategy: pre-filters before any model; bid shading for first-price; pacing and budget on our side
• Failure mode: bidding on everything burns compute and budget; slow responses count as no-bid
• Who pays: our advertisers (overpaying without shading), our infra budget

2.2 When NOT to Build an Ad Serving System#

• You have a handful of direct-sold sponsorships. A scheduling table and a rotation rule — "sponsor A on the homepage this week" — is enough. Auctions earn their complexity with many competing advertisers.
• You want to monetize inventory but advertising is not your business. Join an ad network or exchange and let them run demand, auctions and billing. Your engineering problem becomes an SDK integration and a revenue report.
• You need contextual placements only, no user data. A retrieval-plus-rules system without user-level frequency state and with simpler billing may be enough; the per-user state machinery is the expensive part.
• Your volume is tiny. At 100 requests/s, a database-backed budget check is fine. The lease machinery earns its keep when hot campaigns see thousands of requests per second.

🎯 Staff Insight: "An ad server is a marketplace with a deadline. If there's no marketplace — few advertisers, no competition for slots — I'd use a schedule, not an auction."

2.3 What the Interviewer Leaves Underspecified#

Interviewers deliberately omit:

• Owned inventory vs exchange — the systems barely overlap
• What the advertiser pays for — impressions, clicks or conversions change ranking, billing and fraud exposure
• The budget promise — exact daily cap vs average daily with monthly ceiling
• What counts as an impression — served, rendered or viewable
• Who owns pricing changes — engineering, ads product or finance
• Time zones — budgets "per day" in the advertiser's zone create synchronized resets

Staff engineers surface these and commit. Senior engineers assume them away and get surprised by the follow-up.

2.4 Precise Terminology#

🎯 Staff Insight: If the interviewer says "enforce the budget exactly", ask: "Exactly per request, or exactly on the invoice? We can always bill no more than the budget. Serving no more than the budget, per request, at this QPS, needs a global lock on hot campaigns. I'd bound serving overspend and cap the invoice."

3. Where the Design Splits#

Every ad-serving decision has a technical side (index layout, counter consistency, timeouts) and an economic side (what advertisers pay, what the platform gives away, what users tolerate). Interviewers grade the second side.

3.1 Fault Line 1: Second-Price vs First-Price Pricing#

The tension: In a second-price rule the winner pays roughly what was needed to beat the next bidder, which lets advertisers bid their true value. In first-price the winner pays its bid, which is easier to explain and audit but pushes bidders to shade, making bids noisier and the bidder's tooling more complex.

Staff default: "On owned inventory with many small advertisers, generalized second price with a quality term and experiment-set reserves. It's forgiving to advertisers without bidding tools. If we sell into programmatic exchanges, we follow first price there and offer auto-bidding to advertisers who don't want to shade. Either way the rule is written down per placement."

When to deviate:

• Selling through exchanges: first price is the norm; fighting it costs participation.
• Mostly auto-bid advertisers: the pricing rule matters less because the platform bids on their behalf; focus on target-cost accuracy.
• Guaranteed contracts: fixed-price reserved delivery, served ahead of the auction with its own pacing.

🧭 Principal Move: "A pricing-rule change rewrites every advertiser's strategy. I'd stage it per placement with a holdout, publish it a quarter ahead, and judge it on advertiser ROI and retention — a revenue bump that churns small advertisers isn't a win."

❌ Common L5 Trap: "Second price is always better because it's truthful." Truthfulness holds in the simplest single-slot setting; with multiple slots, quality scores, reserves and auction layers, it isn't strictly true, and the industry largely moved programmatic to first price for transparency. The Staff answer picks a rule for a reason and names who it favors.

3.2 Fault Line 2: Budget Enforcement — Strong Counter vs Leased Slices vs Async#

The tension: An exact budget needs a single, strongly consistent counter per campaign touched on every win. A hot campaign can win tens of thousands of auctions per second, and spend is only known after render. Strong counters melt; async counters lag by minutes.

Staff default: "Leases. The budget service holds the authoritative remaining budget per campaign and grants leases of roughly 10 seconds of a server's expected spend. Servers spend locally. Near exhaustion, lease size shrinks so the tail of the budget is spent in small pieces. Leases expire after 60 seconds and unused amounts return. Overdelivery bound = sum of outstanding leases + impressions in flight between serve and render; I'd report it per campaign and alert above 1%."

When to deviate:

• Low-QPS campaigns: a strong counter is fine; leases are for hot ones. A hybrid routes by campaign QPS.
• Guaranteed delivery contracts: underdelivery is the risk, not overspend — pacing matters more than enforcement.

🧭 Principal Move: "The overdelivery bound is a line item. At our volume, 0.5% overdelivery on $10M daily spend is $50K a day of free inventory. I'd show finance that number next to the cost of tightening it — smaller leases mean more budget-service QPS — and let them choose."

❌ Common L5 Trap: "Use Redis INCRBY on the campaign's spend key for every impression and stop when it crosses the budget." Correct-looking, but it puts a network round trip on the hot path, turns the top campaigns into hot keys at tens of thousands of writes per second, and still counts serves rather than renders. See Hot Keys.

3.3 Fault Line 3: Funnel Depth vs the Latency Budget#

The tension: Scoring more candidates with a heavier model raises relevance and revenue. Every extra millisecond eats a fixed budget, and fan-out amplifies tail latency. Past some point, more scoring loses more impressions to deadline misses than it gains in relevance.

Staff default: "Two-stage ranking under absolute deadlines. Each stage knows the time remaining. Heavy-model scoring stops at its cutoff and the auction uses light-model scores for the rest. Retrieval shards are hedged — send to a second replica after the shard's p95 — so one slow shard doesn't set request latency."

When to deviate:

• Search ads with few slots and high value: spend more of the budget on heavy scoring; each slot is worth more.
• Video pre-roll with seconds of buffer: the budget is looser; deeper ranking is affordable.

🧭 Principal Move: "I'd measure the revenue curve: eCPM gain per extra millisecond of ranking against fill loss per millisecond of p99. The optimal depth is where those cross — and it moves every time the model changes."

❌ Common L5 Trap: "Set a 100 ms timeout on each service call." Sequential stages each with their own 100 ms timeout can take 400 ms. Without an absolute deadline propagated through the request, timeouts compound and the client gives up before you answer.

3.4 Fault Line 4: Frequency Caps — Exact vs Approximate, Fail-Open vs Fail-Closed#

The tension: Caps require per-user, per-campaign state read on the hot path for every candidate that might win. Exactness requires synchronous increments on serve; availability requires deciding what to do when the store is slow or down.

Staff default: "Approximate counters, keyed by (user, campaign, day) in a replicated low-latency store, incremented from render events, read in one batch per request for the top ~20 campaigns, with a 1-second local cache per user on each server. On store timeout (5 ms), fail open — but exclude campaigns with a cap of 1, and add a per-server ceiling so no user sees the same campaign more than 5 times from that server during the outage."

When to deviate:

• Regulated categories (e.g., alcohol, gambling) with legal exposure limits: fail closed for those campaigns.
• Logged-out traffic without stable IDs: caps are best-effort; say so in the advertiser docs.

🎯 Staff Insight: "A cap is a user-experience promise and a reach promise. Product owns the policy for what happens on outage; I own making that policy cheap to execute."

❌ Common L5 Trap: "Store the count in the user's database row and increment it when serving." That's a write on every serve to a store sized for profile reads, counting serves that never rendered — and when it's slow, every ad request is slow.

3.5 Fault Line 5: What You Bill On — Served, Rendered or Viewable#

The tension: Server-side counts are complete and immediate but overcount exposure. Client-side events are true exposure but late, lossy, duplicable and forgeable.

Staff default: "Bill on rendered impressions for display and feed formats, on valid first clicks for CPC. Each served ad carries a signed impression ID; the aggregator accepts the first valid beacon per ID within an hour of serve, runs invalid-traffic filters, and writes to the billing ledger keyed by impression ID. Replays of the event log are idempotent."

When to deviate:

• Brand campaigns sold on viewability: add viewability measurement and bill on it.
• Performance campaigns: bill on conversions with a defined attribution window, accepting days of lag before final spend.

🧭 Principal Move: "Billing is the contract advertisers audit. I'd publish the definition of a billable impression, give large advertisers event-level logs, and track the discrepancy rate against third-party measurement as a product metric."

❌ Common L5 Trap: "We log each served ad and charge from that log." It's complete and simple, and it charges for ads nobody saw. The first large advertiser to compare against their own measurement opens a dispute — and the platform refunds the gap.

4. When It Breaks#

4.1 The Budget Blowout — Feedback Lag Meets a Viral Campaign#

t=0:       Campaign C ($40K/day) matches a trending news topic. Win rate 25× baseline.
t=+30s:    Spend aggregator falls behind: Kafka consumer lag 0s → 6 min (partition skew,
           C's events all hash to one partition).
t=+1min:   Budget service grants leases from stale spend. Pacing controller sees
           spend "on plan" because its input is 6 minutes old.
t=+12min:  Real spend $31K. Reported spend $9K.
t=+18min:  Lag recovers. Budget service sees $46K spent. Revokes leases.
t=+19min:  Final spend $46.8K. Overdelivery 17% — $6.8K of free inventory.

Detection: spend.feedback_lag_s (alert > 60 s); campaign.overdelivery_pct (alert > 1%); aggregator.partition_lag_max.

Mitigation: when feedback lag exceeds a threshold, the budget service shrinks all leases by a factor proportional to lag and pacing caps participation for campaigns whose serve-side win rate jumped — the serving tier knows its own win counts even when billing data is late.

Prevention: partition spend events by impression ID, not campaign, then aggregate per campaign in a second stage; budget leases fed by serve-side spend estimates (price × expected render rate) so enforcement doesn't depend on the slow billing path.

Owner: ads serving on-call (budget service), data platform (aggregator).

4.2 Silent Miscalibration — The Model That Cost 6% of Revenue#

t=0:       New pCTR model ships after an offline AUC gain of +0.4%.
t=+1 day:  Ranking order looks fine. But predicted CTR for one vertical is 1.3× observed.
t=+3 days: CPC advertisers in that vertical win more auctions, pay more per impression,
           click rates below prediction → their cost per click is fine, but their win
           rate crowds out CPM advertisers paying real money.
t=+9 days: Finance flags revenue per 1,000 requests down 6% in that vertical.

Detection: model.calibration_ratio{vertical} = sum(predicted clicks) / sum(observed clicks), alert outside 0.9–1.1; revenue per 1,000 requests by vertical against a holdout.

Mitigation: roll back to the prior model; apply a per-segment calibration layer that rescales predictions to match observed rates daily.

Prevention: online launch criteria include calibration by segment and revenue against a holdout, not just offline AUC. A ranking metric can improve while the prices it implies get worse.

Owner: ads ML team (calibration); ads serving (holdout infrastructure).

4.3 Frequency Store Outage — Fail Open or Go Dark?#

t=0:       Frequency store cluster in one region loses a third of its nodes.
t=+5s:     Cap reads p99: 2ms → 400ms. Every ad request blocks on the batched read.
t=+20s:    serve.deadline_miss_rate: 0.3% → 61%. Fill rate collapses in that region.
t=+3min:   On-call flips cap checks to fail-open via feature flag. Fill recovers.
t=+40min:  Users report seeing the same ad 11 times in a session.

Detection: freqstore.read_p99, serve.deadline_miss_rate, serve.capcheck_failopen_total.

Mitigation: cap reads get their own 5 ms deadline and fail open automatically; a per-server, per-user in-memory ceiling (5 impressions per campaign per hour) applies during fail-open; campaigns with cap = 1 are excluded.

Prevention: the fail-open policy is decided in advance by product and coded as the default, not improvised during the incident; chaos tests kill a third of the store monthly.

Owner: ads serving on-call; product owns the fail-open policy.

4.4 The Slow Dependency — Feature Store Tail Eats the Budget#

t=0:       A feature store compaction job runs during peak. p99 5ms → 70ms.
t=+10s:    Ranking waits for features; heavy model starts at t=+75ms of an 80ms budget.
t=+30s:    Deadline misses 0.3% → 28%. Clients render blank slots or house ads.
t=+2min:   Revenue per minute down 25%.

Detection: featurestore.read_p99, rank.stage_budget_exceeded_total, serve.fill_rate.

Mitigation: feature reads get a hard 8 ms cutoff; missing features use defaults so the light model still runs; heavy model skipped if under 10 ms remain.

Prevention: compaction scheduled off-peak with rate limits; hedged reads to a second replica after p95; the degradation ladder tested by injecting latency weekly.

Owner: feature store team (compaction), ads serving (degradation ladder).

4.5 Double Billing After a Pipeline Replay#

t=0:       Aggregator bug drops 2 hours of click events. Fix deployed.
t=+1h:     Data engineer replays the event log from 2 hours back to recover clicks.
t=+1h:     Aggregator's dedupe state only holds 30 minutes of impression IDs.
t=+2h:     1.5 hours of impressions are billed twice. 4,100 advertisers overcharged.
t=+3 days: Advertisers' third-party numbers disagree; disputes open.

Detection: billable impressions per hour vs served-and-rendered from the raw log (should never exceed); billing.duplicate_rejections drops to zero during a replay (suspicious).

Mitigation: recompute billing for the window from raw events with full dedupe; issue credits.

Prevention: the billing ledger enforces uniqueness on impression ID itself — an idempotency key at the sink — so replays are safe regardless of the stream processor's dedupe window. See Idempotency & Exactly-Once.

Owner: billing team (ledger), data platform (replay runbook).

4.6 The Midnight Herd — Budgets Reset at Once#

At 00:00 in a major time zone, thousands of campaigns that exhausted yesterday's budget reset simultaneously. Every ad server requests fresh leases in the same second; the budget service sees 400 servers × 30K campaigns of lease requests, and its p99 jumps to seconds. Campaigns serve nothing for the first few minutes of the day, then pacing — seeing them behind plan — overcorrects and spends aggressively.

Detection: budget.lease_request_rate spike at time-zone boundaries; campaign.zero_serve_minutes_after_reset.

Prevention: pre-issue first leases for the new day a few minutes before reset with a "valid from" timestamp; jitter lease refreshes; pacing plans start from the first minute, not from the first observed spend.

Owner: ads serving (budget service).

4.7 Operational Reality Matrix#

🎯 Staff Insight: The two numbers that matter are on different sides of the ledger. "serve.deadline_miss_rate tells me we're losing revenue; campaign.overdelivery_pct tells me we're giving it away. I'd page on both, and neither is visible in average latency."

5. Scorecard#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Deep ML knowledge ≠ ad serving design. A beautiful ranking model doesn't bound overspend or dedupe beacons.
• Auction theory ≠ marketplace judgment. Reciting incentive properties without naming who the rule favors and how to change it is incomplete.
• "We use Redis" ≠ budget enforcement. A fast counter is still a hot key and still counts the wrong event.
• Low average latency ≠ meeting the deadline. The fill rate lives at p99 across a fan-out.

6. The 45 Minutes, Phase by Phase#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Model architecture — "light model then heavy model; I'd focus on calibration."
• Creative review and policy — one line: ads are reviewed before entering the index.
• Targeting taxonomy — "inverted index over attributes."
• Advertiser UI — say what the delivery report shows, not how.
• Attribution modeling — mention the window; skip the math.

6.4 Follow-Up Questions to Expect#

1. "A campaign's win rate jumps 30×. Minute by minute, what happens to its budget?"
2. "Split the 100 ms. What happens when one stage is slow?"
3. "Second price or first price? What do advertisers do differently under each?"
4. "The frequency store is down. Serve or not? Who decided?"
5. "What exactly is a billable impression, and how do you prevent double billing on replay?"
6. "How do you know a new model isn't quietly losing money?"
7. "Budgets are per day in the advertiser's time zone. What happens at midnight?"

7. Practice Rounds#

Drill 1: The Opening#

Prompt: "Design an ad serving system for our app."

Drill 2: Split the Latency Budget#

Prompt: "You have 100 ms end to end. Walk me through where it goes and what happens when a stage is slow."

Drill 3: Make It Concrete — Size the Funnel and the Counters#

Prompt: "Size the serving tier, the frequency store and the event pipeline for 500K requests per second."

Drill 4: The Viral Campaign#

Prompt: "A $50,000 daily campaign starts winning 30× more auctions at 9 a.m. How much does it overspend, and why?"

Drill 5: Dependency Down — The Frequency Store#

Prompt: "The frequency-cap store in one region just became slow — p99 400 ms. What happens, and what should happen?"

Drill 6: The Hot Campaign#

Prompt: "One advertiser's campaign is eligible for 40% of all requests. What breaks?"

Drill 7: Build vs Buy#

Prompt: "Should we build our own ad serving stack or use a third-party ad server and exchange?"

Drill 8: Changing a Reserve Price Without an Outage#

Prompt: "Product wants to raise reserve prices on the home feed by 30%. How do you ship it?"

Drill 9: The Cost Question#

Prompt: "Where does the money go in this system, and how would you cut infrastructure cost by 30%?"

Drill 10: Multi-Region Budgets#

Prompt: "We serve from four regions. How do budgets work across them?"

8. Incident Walkthroughs#

Deep Dive 1: Peak-Traffic Incident — The Championship Game#

Context: During a major sports final, ad requests jump from 300K/s to 1.4M/s in the 90 seconds after kickoff. Fill rate drops from 96% to 58%, and the top brand campaigns — which paid for this moment — are underdelivering. The on-call escalates to you.

Questions to Surface First:

• Are we missing deadlines (latency) or declining to serve (budget, pacing, caps)?
• Which stage is over its budget — retrieval, features, ranking or guard?
• Are leases being refused because the budget service is overloaded?
• Is pacing throttling brand campaigns that are behind plan because the plan didn't expect this spike?

Typical L5 Approach: Scales the serving fleet. Autoscaling takes 6 minutes; by then the spike has moved to halftime and the brand campaigns have underdelivered.

Staff Approach: Reads stage metrics: ranking is fine, but the budget service's lease-refill p99 is 900 ms under the refill surge, so guards fall through to lower candidates. Raises lease sizes for the top campaigns (a config already built for this), which cuts refill QPS ~5×. Pacing plans for brand campaigns flagged as event-driven get a spike-shaped curve.

Principal Approach: Treats scheduled marquee events as capacity events with a playbook: pre-scaling, pre-sized leases, pacing curves for event-driven campaigns, and a sales process that tells the platform which campaigns bought the moment.

Key Takeaway: "Lease size must scale with spend rate, or the budget service becomes the bottleneck exactly when the money is largest."

What clears the Staff bar:

• Distinguishes deadline misses from declines before acting
• Fixes lease sizing rather than just adding servers
• Connects the incident to how campaigns are sold

Deep Dive 2: Silent Failure — Revenue Down 4%, Nothing Alerting#

Context: Finance reports that revenue per 1,000 requests has drifted down 4% over two weeks. Latency, fill rate and error rates are all normal. Nobody changed the auction.

Questions to Surface First:

• Did a model, feature or reserve-price config change in the window?
• Is the drop uniform or concentrated in a segment — vertical, device, region?
• Are predictions still calibrated by segment?
• Did invalid-traffic filtering change, shifting what counts as billable?

Typical L5 Approach: Checks dashboards, sees all-green service health, and attributes it to seasonality.

Staff Approach: Splits revenue per 1,000 requests by segment and finds the drop is concentrated in Android. A feature pipeline change two weeks earlier started sending a device feature as null for a new OS version; the model's predictions for those users regressed toward the mean, compressing eCPM and clearing prices. Backfills the feature and adds per-feature null-rate monitoring.

Principal Approach: Establishes that revenue per 1,000 requests by segment, against a long-running holdout, is a first-class production signal — the same rank as latency — with an owner and an alert.

Key Takeaway: "In ads, the most expensive failures don't throw errors. They show up as prices."

What clears the Staff bar:

• Segments the revenue signal instead of accepting averages
• Traces the drop to a feature, then to calibration
• Adds monitoring on inputs, not just outputs

Deep Dive 3: Large-Customer Onboarding — 80,000 Campaigns and a 100M-User Audience#

Context: A large retailer is onboarding with 80,000 campaigns (one per product category and region), each targeting a custom audience list of up to 100 million user IDs, with tight frequency caps. Sales has signed. You're asked to make it work.

Questions to Surface First:

• How are custom audiences represented in the targeting index — and how big are they?
• What does 80,000 more campaigns do to retrieval posting lists and pacing controller load?
• How many of these campaigns compete with each other for the same users?
• What frequency-cap level does the retailer want — per campaign or per advertiser?

Typical L5 Approach: Loads audiences as user-ID lists attached to each campaign. Retrieval now intersects 100M-entry lists per request; p99 retrieval doubles.

Staff Approach: Inverts the representation: user → audience-segment IDs, stored with user features, so retrieval reads the user's segments (tens) and looks up campaigns targeting those segments. Proposes advertiser-level frequency caps so the retailer's own campaigns don't saturate one user. Groups campaigns under a shared budget so pacing runs on 50 budget groups, not 80,000 campaigns.

Principal Approach: Recognizes a product gap: large advertisers need campaign groups with shared budgets and advertiser-level caps. Productizes both, with limits on audience-list size per plan.

Key Takeaway: "Invert large audiences to user-side segments, and pace groups, not campaigns."

What clears the Staff bar:

• Finds the representation problem in targeting before it hits retrieval latency
• Prevents an advertiser from competing with itself
• Reduces control-loop load with budget groups

Deep Dive 4: Post-Mortem — $410,000 in Duplicate Charges#

Context: After a stream-processor upgrade, a replay of 3 hours of events was used to recover a gap. Billing double-counted 2.5 hours of impressions for 4,100 advertisers. Advertisers' third-party measurement disagreed by ~14%, and disputes followed. You're leading the post-mortem.

Questions to Surface First:

• Where was deduplication enforced — the stream processor's state, or the ledger?
• How long was the dedupe window, and how long was the replay?
• Did any alert compare billable counts to raw unique impression IDs?
• Who approved the replay, and was there a runbook?

Typical L5 Approach: Increases the stream processor's dedupe window to 6 hours.

Staff Approach: Moves the uniqueness guarantee to the sink: the billing ledger rejects a second write with the same impression ID, regardless of what the stream processor does. Adds a reconciliation job comparing billable impressions per hour to unique rendered impression IDs from raw events; any excess pages. Issues credits computed from the raw log.

Principal Approach: Establishes that every pipeline writing money has idempotency at the sink and an independent reconciliation, and that replays of money-bearing streams require a reviewed runbook. Publishes the definition of billable impressions to advertisers.

Key Takeaway: "Dedupe in the stream is an optimization. Uniqueness in the ledger is the guarantee."

What clears the Staff bar:

• Moves idempotency to the sink rather than tuning a window
• Adds independent reconciliation against raw events
• Treats replays of money streams as controlled operations

Deep Dive 5: Multi-Region Expansion — Serving EU Users From the EU#

Context: The company is moving EU ad serving into an EU region for latency and data-protection reasons. User-level data for EU users — features, frequency counters, event logs — must stay in the EU. Advertisers run global campaigns with a single budget.

Questions to Surface First:

• Which state is user-level (must stay) and which is campaign-level (can be global)?
• How do global budgets work when one region's spend must be reported without user data?
• How does consent affect what signals can be used for ranking?
• What happens to models trained on global data?

Typical L5 Approach: Deploys a copy of the serving stack in the EU and replicates all databases between regions.

Staff Approach: Splits state: campaign data and budgets are global, replicated to the EU; user features, frequency counters and raw events for EU users live only in the EU. The EU region reports aggregated spend per campaign — no user IDs — to the global budget coordinator. Ranking uses only signals permitted by each user's consent state, with a contextual-only model path when consent is absent.

Principal Approach: Defines data classification — user-level vs aggregate — as a company standard, with automated checks that no user-level ads data crosses regions, and a model governance process for training on regional data.

Key Takeaway: "Budgets are global, users are local. Only aggregates cross the border."

What clears the Staff bar:

• Classifies state by residency requirement before moving anything
• Keeps dedupe in-region so only aggregates cross
• Handles consent as a ranking-path decision

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Split a ~100 ms budget into per-stage deadlines and define a degradation ladder that always returns a valid response
• Design a two-stage retrieval and ranking funnel and size it by scoring work
• Rank by eCPM with a quality term and explain how second- and first-price rules change advertiser behavior
• Enforce budgets with leased slices and derive the overdelivery bound
• Pace spend with a throttling controller and explain why feedback lag is the main risk
• Implement approximate frequency caps that fail open with a ceiling
• Define billable impressions and make billing idempotent at the ledger
• Detect silent revenue regressions through calibration and holdouts

The Bar for This Question#

Mid-level (L4): Builds a service that filters ads by targeting, sorts by bid, returns the top one and logs it. Works in a demo. No deadline, no budget story, bills on serve.

Senior (L5): Adds an ML model for CTR, ranks by bid × pCTR, checks budget in a database, caps frequency with counters, and logs impressions to Kafka. The gap: synchronous budget checks on hot campaigns, no deadline propagation, no named pricing rule, billing on served ads, and exact caps on the hot path. The design works at average load and fails on the first viral campaign or slow dependency.

Staff+ (L6): Frames the problem as spending other people's money under a deadline within the first five minutes. Propagates an absolute deadline with a degradation ladder. Names and justifies the pricing rule. Bounds overdelivery with leases and paces with a controller, aware of feedback lag. Caps frequency approximately with an owned fail-open policy. Bills on rendered, deduplicated, filtered events with sink-level idempotency. Names who pays — the platform absorbs bounded overdelivery; advertisers never pay for unrendered or duplicate impressions. The interviewer should learn something from the answer.

10. Hot Takes#

10.1 "Exact Budgets" Are a Billing Feature, Not a Serving Feature#

The Staff position: Cap the invoice exactly; bound serving overdelivery and report it.

Why this matters in interviews: Promising exact serving-time budgets is a Senior signal; computing the bound is Staff.

10.2 Calibration Matters More Than AUC#

The Staff position: Gate launches on calibration by segment and revenue against a holdout.

Why this matters in interviews: It shows you know predictions are prices, not just scores.

10.3 Billing on Serve Is Overbilling With Extra Steps#

The Staff position: Bill only the third number, with idempotency at the ledger.

Why this matters in interviews: Interviewers ask "what do you bill on?" to see if you know there's a choice.

10.4 A Late Ad Is Worth Less Than a Worse Ad#

The Staff position: Design the degradation ladder before tuning the model.

Why this matters in interviews: It reorders priorities from relevance to deadline, which is how production ad systems actually behave.

10.5 The Pricing Rule Is a Product Decision#

The Staff position: Second- vs first-price isn't an algorithm choice. It decides whether advertisers or the platform does the bid optimization, and changing it is a migration for every buyer. Engineers implement it; ads product and finance own it.

Why this matters in interviews: Treating pricing as a config flag is the clearest sign of missing marketplace judgment.

11. Beyond Staff: The Principal View#

Why L7 Sees This Problem Differently#

The Staff engineer builds a serving system that meets its deadline, bounds overspend and bills correctly. The Principal engineer notices that feed ads, search ads and video ads each run their own auction, their own pacing and their own billing — and that an advertiser running on all three gets three budgets that don't know about each other, three definitions of an impression and three dashboards. The L7 problem is one marketplace: shared budget and pacing across surfaces, one billing definition, one auction framework with per-surface parameters, and governance for pricing changes.

The Org-Level Fault Line#

One ads platform vs per-surface ad stacks.

🧭 Principal Move: "The platform owns everything that touches money: budgets, pacing, auction framework, billing and invalid-traffic filtering. Surfaces own retrieval and ranking for their format and plug in via a scored-candidate interface. No surface bills on its own."

Cost Model#

Assumptions: CPU serving with batched inference for light models, accelerators for heavy models at large scale; event pipeline on a managed log; fully loaded engineer ~$250K/year. Illustrative ranges.

The pricing insight: at scale, infrastructure is a small fraction of revenue, so the binding costs are revenue errors — miscalibration, deadline misses, overdelivery and disputes. A 0.5% overdelivery rate on $10M/day of spend is ~$1.8M a year; a 1% eCPM loss from an over-aggressive cost cut is larger than most infra budgets.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-ADS-001: Money-Bearing Ad Decisions
Status: Approved   Owners: Ads Platform + Finance

Scope
  Every system that selects, prices or bills an ad on any surface.

MUST
  1. Price through the shared auction framework; pricing rules are documented
     per placement and changed only through the pricing-change process.
  2. Enforce budgets via the platform budget service; report overdelivery per
     campaign; alert above 1% of daily budget.
  3. Bill only rendered, deduplicated, invalid-traffic-filtered events, with
     uniqueness on impression ID enforced at the ledger.
  4. Propagate an absolute request deadline; return a valid response or a no-ad
     signal before it.
  5. Gate model launches on calibration by segment and revenue vs holdout.

SHOULD
  1. Use approximate frequency caps with a product-approved fail-open policy.
  2. Size budget leases from short-window spend rates.
  3. Keep a 1-5% long-running holdout per surface.

Exceptions
  Filed with Ads Platform; finance sign-off required for any billing exception;
  time-boxed to one quarter.

Success metrics
  - Overdelivery: under 1% of spend platform-wide
  - Billing discrepancy vs third-party measurement: within agreed tolerance
  - Deadline miss rate: under 0.5% at p99 traffic
  - Surfaces billing outside the platform: 0

What I'd Tell the VP#

"Our three ad surfaces each run their own budgets and billing, so an advertiser spending on all three gets three invoices with three definitions of an impression, and we can't pace their spend across surfaces. I'm proposing a shared platform for everything that touches money — budgets, auctions, billing — while surface teams keep ownership of ranking. It's roughly a year of a 10-person team. The return is fewer billing disputes, cross-surface campaigns that sales can sell as one product, and overdelivery we can measure and cap instead of discovering. The main risk is migration; I'd start with billing, where inconsistency costs us most."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll build a great auction for the feed" — correct, but ignores the other surfaces with their own budgets.
• "Overdelivery is bounded under 1%" — good, but not translated into dollars or agreed with finance.
• "We'll add an EU region" — no data classification separating user-level from aggregate state.

Appendices

Appendix A: Mechanics in Depth#

A.1 The Serving Path With Deadlines#

handle(request):
  deadline = now() + 80ms
  user = resolve_user(request, deadline - 70ms)           # cached, ~1ms
  cands = retrieval.query(targeting(user, request.context),
                          exclude=pacing.throttled_sample(),
                          deadline=deadline - 60ms)        # hedged across shards
  if cands.empty: cands = placement_cache.recent_winners(request.placement)
  feats = features.batch_get(user, cands, deadline=deadline - 50ms) or defaults
  light = light_model.score(cands, feats)
  top = top_k(light, 300)
  heavy = heavy_model.score(top, feats, deadline=deadline - 15ms)  # partial ok
  scored = merge(heavy, light)                            # light where heavy missing
  for winner, price in auction.run(scored, request.placement):
      if not freq.allows(user, winner.campaign, deadline=deadline - 5ms): continue
      if not leases.try_spend(winner.campaign, price):    continue
      imp = sign(impression_id(), winner, price)
      log.served(imp)
      return ad(winner, imp)
  return NO_AD   # HTTP 204

A.2 Pricing Rules#

eCPM(ad) = bid_cpm                          for CPM
         = bid_cpc * pCTR * 1000            for CPC
         = bid_cpa * pCTR * pCVR * 1000     for CPA
score(ad) = eCPM(ad) * quality(ad)

Generalized second price (one slot):
  winner = argmax score
  price_eCPM = max(reserve, score(runner_up) / quality(winner)) + 0.01
  CPC advertisers are charged price_eCPM / (pCTR * 1000) per click

First price:
  price_eCPM = eCPM(winner)  (bidder shades; platform may offer auto-bidding)

A.3 Pacing Controller#

every 30s for each campaign c:
  planned = budget(c) * traffic_curve_fraction(now, c.tz)    # share of day elapsed
  actual  = spend_estimate(c)          # serve-side estimate, corrected by billing
  error   = (actual - planned) / budget(c)
  p(c)    = clamp(p(c) * exp(-k * error), p_min, 1.0)        # multiplicative update
  if remaining(c) < 5% of budget: lease_size(c) = small
serving: campaign c enters an auction with probability p(c)

Appendix B: Data Model#

CREATE TABLE campaigns (
  campaign_id    TEXT PRIMARY KEY,
  advertiser_id  TEXT NOT NULL,
  budget_group   TEXT,                       -- shared budget across campaigns
  daily_budget   BIGINT NOT NULL,            -- micros
  budget_tz      TEXT NOT NULL,
  bid_type       TEXT NOT NULL,              -- cpm | cpc | cpa
  bid_micros     BIGINT NOT NULL,
  freq_cap       INT,                        -- impressions per user per day
  targeting      JSONB NOT NULL,
  state          TEXT NOT NULL               -- active | paused | exhausted
);

-- billing ledger: uniqueness is the double-billing guard
CREATE TABLE billable_events (
  impression_id  TEXT PRIMARY KEY,
  campaign_id    TEXT NOT NULL,
  event_type     TEXT NOT NULL,              -- impression | click
  price_micros   BIGINT NOT NULL,
  rendered_at    TIMESTAMPTZ NOT NULL
);

-- frequency store (KV): key = user_id|campaign_id|yyyymmdd, value = count, TTL 26h

Appendix C: Coordination Mechanisms#

C.1 Budget Leases#

C.2 Quick Comparison#

Appendix D: API Contract & Client Behavior#

• Clients send deadline_ms and render a house ad on 204 or on timeout; they never retry an ad request synchronously.
• The impression beacon fires once on render; retries on network failure are allowed because billing dedupes by impression ID.
• Beacons older than 1 hour after serve are logged but not billed.
• Click URLs redirect with a 302 in under 20 ms; click logging is async.
• Exchange-style integrations follow the OpenRTB contract: honor tmax, return HTTP 204 for no-bid.

Appendix E: Observability#

Core metrics:

• serve.deadline_miss_rate, serve.fill_rate, rank.stage_budget_exceeded_total{stage}
• campaign.overdelivery_pct, spend.feedback_lag_s, budget.refill_p99, guard.fallthrough_rate
• model.calibration_ratio{segment}, revenue_per_mille{segment} vs holdout
• freqstore.read_p99, serve.capcheck_failopen_total
• billing.reconciliation_excess, billing.duplicate_rejections, ivt.filtered_rate

Critical alerts:

Debugging the silent failure: service health is useless for revenue regressions. Watch revenue per 1,000 requests by segment against a holdout, calibration by segment and feature null rates.

Appendix F: Scale Evolution#

What you don't build on day one: auto-bidding, cross-surface budgets, multi-region budget coordination, viewability billing, advertiser audit logs. Each has a trigger in Section 11.

Appendix G: Multi-Tenancy, Fairness & Cost#

• Advertiser fairness: one ad per advertiser per slot by default, so large advertisers don't bid against themselves or crowd out small ones with near-duplicate campaigns.
• Budget groups: large advertisers pace groups, not individual campaigns, to keep controller load bounded.
• Small-advertiser health: track win rate and retention by spend tier; pricing changes are judged on it.
• Cost attribution: inference cost per surface and per model; model launches report cost per 1,000 requests alongside revenue impact. Use the back-of-envelope calculator and latency budget tool to rehearse the sizing.