Technologies referenced in this case study: Time-Series Databases · Kafka · Flink · Cassandra · Redis · Elasticsearch

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once, then return to sections for targeted review.

Without cardinality controls:
t=0:       A team adds tag customer_id to http.request.duration (2M customers)
t=+10min:  Active series for one tenant: 1.2M -> 38M
t=+15min:  Ingest shard memory 60% -> 97%; index inserts back up
t=+18min:  Shared shard OOMs; 400 other tenants on it lose ingestion
t=+20min:  Alert evaluator reads empty series; "no data" alerts disabled by default
t=+45min:  A real outage at a different customer goes unpaged
t=+2h:     Postmortem: the monitoring system was the blast radius

With per-tenant cardinality limits:
t=0:       Same deploy, same tag
t=+2min:   New-series rate for the tenant exceeds 50K/min; limiter engages
t=+2min:   Excess series dropped at intake; tenant notified with the offending tag
t=+5min:   Shared shards unaffected; every other tenant's alerts keep evaluating
t=+1h:     Team switches customer_id to a log/trace attribute; metric reverts

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What This Interview Actually Tests#

A metrics platform is not a time-series database question. Everyone can describe a TSDB.

It is a cost-of-cardinality and trust-in-alerts question that tests:

• Whether you identify active series — not bytes — as the unit of memory, cost, and failure
• Whether you design the alerting path to be more reliable than the systems it watches, with explicit behavior for late and missing data
• Whether you isolate tenants so one bad tag can't blind everyone
• Whether you connect resolution, retention, and query flexibility to a price someone pays

The key insight: The monitoring system is the one piece of infrastructure that must keep working during everyone else's outage — when traffic, cardinality, and query load all spike at once. Staff engineers design for that moment: the write path protects the alert path, the alert path protects itself from the query path, and every tenant has a cardinality budget.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll design this as a multi-tenant SaaS like Datadog, and I'll treat the alerting path as the product. Dashboards and long-term analytics matter, but if we're slow on a dashboard, a user is annoyed; if we miss a page, a customer's outage goes unnoticed. So the write path protects the alert path, and cardinality limits protect the write path."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Citing specific, publicly documented monitoring systems shows you've studied the real constraints — cardinality, compression, and global query.

Facebook Gorilla — Compression That Changed the Memory Math#

Facebook's Gorilla paper (VLDB 2015) described an in-memory TSDB for the most recent ~26 hours of data using delta-of-delta timestamp encoding and XOR float compression, getting from 16 bytes to ~1.37 bytes per point on average. That compression let them keep recent data in memory for fast queries, with an on-disk store behind it.

Staff insight: Once points cost ~1.4 bytes, bytes stop being the bottleneck and series overhead dominates. Every modern TSDB inherited this, and it's why cardinality — not volume — is the question.

Google Monarch — Regional Ingestion, Global Query#

Monarch (VLDB 2020) is Google's planet-scale, multi-tenant, in-memory monitoring system. Data is ingested and stored in regional "zones" close to where it's produced, and a global query layer fans out and merges. Monarch pushes aggregation toward the leaves and was explicitly designed so monitoring keeps working regardless of the health of the systems being monitored — for example, it avoids depending on Google's own distributed storage systems for its core path.

Staff insight: "Monitoring must not depend on what it monitors" is a design rule, not a slogan. Say it, and name the dependency you're removing.

Prometheus — Pull, Local Storage, and the Cardinality Warning#

Prometheus (originally from SoundCloud, now a CNCF project) scrapes targets, stores data locally in 2-hour blocks with an in-memory head, and evaluates recording and alerting rules on the same server. Its documentation repeatedly warns against high-cardinality labels like user IDs. Horizontal-scale projects such as Thanos, Cortex/Mimir, and Uber's M3 exist largely to add long-term storage, global query, and multi-tenancy on top of that model.

Staff insight: Prometheus colocates alert evaluation with the scraper — so alerts keep working even if central storage is down. That locality is a reliability feature worth copying in any design.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: Intake enforces limits before anything touches shared state. Kafka decouples intake from storage and lets two independent consumers read the same data: the alert evaluators (hot path, streaming, isolated capacity) and the ingest writers (storage path). Queries hit recent data in the ingest heads and older data in blocks. The watchdog lives outside the platform's failure domain and verifies the whole loop — metric in, page out.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 20 minutes on the storage engine — compression, column layouts, Cassandra schema — and never reach cardinality limits, the alert path, or tenant isolation. Compress storage to 5 minutes. The level is decided by what you protect and in what order.

Phase 1: Requirements & Framing (2–3 minutes)#

Functional scope in one sentence:

"Customers send tagged metrics from agents; we store them, serve dashboards and ad-hoc queries, and evaluate alert rules that notify on-call through pager, chat, or email."

Then the non-functional requirements that shape the design:

"The numbers I'll design around: a billion active series, and a two-minute metric-to-page SLO that must hold during a customer's worst day — which is exactly when their cardinality and query load spike."

Phase 2: Core Entities & API (1–2 minutes)#

Series      { tenant_id, metric_name, tags: sorted map, series_id = hash(tenant, name, tags) }
Point       { series_id, timestamp (s or ms), value (float64) | sketch }
Monitor     { monitor_id, tenant_id, query, window, threshold(s), eval_delay,
              no_data_policy, for_duration, notify_targets, owner_team }
AlertState  { monitor_id, group_key, state: OK | WARN | ALERT | NO_DATA, since, last_eval }

APIs:

POST /v1/series            batch of {metric, tags[], points[]}   -> 202 (async)
POST /v1/distribution      batch of sketches                     -> 202
GET  /v1/query?q=avg:http.latency{service:checkout} by {region}&from=&to=
POST /v1/monitors          create/update rule
GET  /v1/monitors/{id}/state

"The identity decision that matters: a series is tenant + metric + the full sorted tag set. Every distinct tag combination is a new series with its own memory cost — that's the definition of cardinality."

Phase 3: High-Level Architecture (≤5 minutes)#

Staff candidates spend under 5 minutes here. Draw two consumers off one log.

Say it in four sentences:

1. "Agents pre-aggregate locally and push batches every 10–15 s; intake authenticates, stamps the tenant, and enforces rate and cardinality limits before anything reaches shared state."
2. "Kafka is the durable buffer: partitioned by tenant and series hash, 24–72 h retention, so storage can fall behind or replay without losing data."
3. "Two independent consumers: alert evaluators consume the stream directly and keep windowed state in memory; ingest writers build compressed in-memory heads and flush 2-hour blocks to object storage with a tag index."
4. "Queries fan out to heads for recent data and to blocks or rollups for older data, with per-query cost limits."

Phase 4: Transition to Depth (1 minute)#

"Storage engines for time series are well understood; I'll use the Gorilla-style design. The three places I'd like to go deep: cardinality — how we stop one tenant's tag from taking down shared infrastructure; the alerting path — latency, late data, no-data semantics, and isolation from queries; and multi-tenant isolation and cost. Which first?"

Default order: cardinality → alerting → tenancy/cost.

Phase 5: Deep Dives (25–30 minutes)#

Deep dive 1 — Cardinality (8–10 min).

• Cost per series: head chunk + index postings + series metadata ≈ 2–8 KB of RAM, plus query fan-out cost.
• Limits at intake, per tenant: active series cap (contracted), new-series-per-minute cap (e.g., 50K/min), per-metric series cap (e.g., 100K), tag value length caps.
• When a limit trips: drop new series (never existing ones — existing series back alerts), emit a tenant.cardinality_limited event naming the metric and tag, notify the tenant.
• Detection of the culprit: per-metric HyperLogLog of series and per-tag distinct-value estimates so we can say "tag customer_id on http.request.duration added 3.4M series."
• Tooling: allow tenants to configure tag allowlists per metric (keep service, endpoint, drop pod_id) — aggregation at intake reduces series before storage.

Deep dive 2 — Alerting path (8–10 min).

• Evaluators consume Kafka partitions; each owns the monitors whose series hash to its partitions (or evaluates tenant-wide queries via a pre-aggregation stage).
• Windowed state: for avg(last_5m) > 500ms, keep per-group running sums/counts per 10 s bucket.
• Evaluation delay: wait 30–60 s past window end before evaluating, because agents flush every 10–15 s and pipelines have jitter.
• No-data policy per rule: alert, resolve, or keep last state, with a no-data timeout (e.g., 10 min). Default for infrastructure monitors: alert on no data.
• Hysteresis: for: 5m and separate recovery thresholds to prevent flapping.
• Notifier: dedup by (monitor, group), group notifications, escalation, and delivery confirmation — a page that fails to send is retried via a second provider.

Deep dive 3 — Tenant isolation and cost (6–8 min).

• Cells: each cell is a full stack (intake shards, Kafka cluster, writers, evaluators) serving a subset of tenants; the largest tenants get dedicated cells.
• Query cost limits: max series scanned per query (e.g., 500K), max points returned, timeouts, and per-tenant concurrent query slots.
• Billing by the cost driver: custom metric series (per month), ingested points or hosts, and retention tier.

Phase 6: Wrap-Up (2–3 minutes)#

"To summarize: cardinality is the unit of cost and failure, so limits live at intake; two independent consumers off a durable log, with alert evaluation isolated from queries; tiered storage with rollups and sketches; cells for blast radius; and a watchdog outside our failure domain. Next I'd build per-tenant cardinality explorers so customers can fix their own tag explosions, SLO-based alerting (burn rates) to replace threshold alerts, and cost attribution by team inside each customer."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Monitoring is infrastructure that people trust implicitly. An engineer who sees no alert assumes nothing is wrong. That trust makes silent failures in the monitoring system uniquely dangerous — and makes this a clean test of whether a candidate designs for the failure mode users can't see.

It's also the purest cardinality problem in system design. Every other system has a natural bound on its keys; a metrics system's keys are whatever tag combinations its users invent. Candidates who don't notice that one line of instrumentation code can multiply storage cost by 1,000× haven't operated one.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"During the customer's worst outage, which of our paths is still guaranteed to work?"

The customer's worst day is our worst day: their error metrics spike, their autoscaler adds 5,000 hosts (new series), their engineers open 300 dashboards, and they create ad-hoc queries over 30 days of data. Every load type spikes at once. If the answer is "everything shares the same TSDB," the design fails when it matters. The Staff answer names the protected path (ingestion of existing series → alert evaluation → notification) and what's shed first (ad-hoc long-range queries).

🎯 Staff Move: "I want an explicit shed order. Under overload we drop ad-hoc queries first, then dashboard refresh rates, then long-range rollup reads. Ingestion of existing series, alert evaluation, and notification delivery are the last things we give up."

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Intent 1 — Operational alerting. The job is to tell a human that something is wrong within ~1–2 minutes, and never to silently stop watching. The design priorities are freshness, isolation, redundancy, and precise semantics for late and missing data. Everything else in the platform exists to support this path or is secondary to it.

Intent 2 — Exploration and dashboards. Engineers slice metrics by tags during incidents and reviews. The priorities are flexible filtering and grouping over high-cardinality tags with interactive latency. The risk is the "query of death" — a regex over a tag with 10M values across 30 days.

Intent 3 — Long-term analytics. Capacity planning, year-over-year trends, SLO reporting. The priorities are cheap storage and correct aggregates at coarse resolution. The risk is subtle wrongness: averaging percentiles, or a counter reset turning into a negative rate.

2.2 When NOT to Build This#

🎯 Staff Move: "The single most useful rule I'd publish: if the tag's values are unbounded — user IDs, request IDs, URLs with parameters — it's a log or trace attribute, not a metric tag."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: Cardinality — Expressiveness vs Cost#

The tension: Every tag makes queries more powerful and multiplies series. Users want to slice by anything; the platform pays per series in RAM, index, and query fan-out.

Staff default: Rate limit on new series plus a contracted active-series cap, with existing series always accepted; per-metric cardinality caps; tag allowlists configurable per metric; and a cardinality explorer that names the offending tag within minutes. Autoscaling churn gets headroom: the new-series limit is sized to absorb a 2–3× fleet scale-out.

🎯 Staff Move: "When a tenant hits a limit, I drop new series, never existing ones. Existing series are what their alerts are watching."

3.2 Fault Line 2: Push vs Pull Ingestion#

Staff default: Hybrid. The agent scrapes local Prometheus-style endpoints and integrations, aggregates, and pushes to intake every 10–15 s, with a disk buffer (e.g., 15–60 min) during network failures. The agent emits its own heartbeat series, so "host dead" vs "pipeline broken" is distinguishable: a missing heartbeat from one host = host issue; missing heartbeats from 10,000 hosts at once = our issue.

When to deviate: An internal platform inside one network can scrape centrally (Prometheus model) and gains simpler liveness.

3.3 Fault Line 3: Alert Freshness vs Completeness#

The tension: Evaluate a window as soon as it closes and you alert on partial data (false positives on ratios, false negatives on sums). Wait for stragglers and every page is slower.

Staff default: Fixed evaluation delay (e.g., 60 s for 10 s data) as a per-rule default, configurable down for metrics from low-latency sources; the pipeline tracks per-partition watermarks and exposes alerting.eval_lag so the delay can be tuned from data. Ratios are computed only when both numerator and denominator windows are complete.

3.4 Fault Line 4: Shared vs Isolated Tenancy#

Staff default: Cells, each serving a few hundred to a few thousand tenants, with shuffle sharding inside a cell and dedicated cells for the top ~1% of tenants by series. Cell count is a lever: more cells = smaller blast radius and more operational overhead. Deploys roll out cell by cell.

3.5 Fault Line 5: Resolution vs Retention#

Staff default: Raw 10 s for 15 days; 1 m rollups (sum/count/min/max; sketches for distributions) for 15 months; query planner picks resolution by range (e.g., > 2 days → 1 m, > 60 days → 1 h derived on read). Counters are stored as rates or deltas with reset handling at ingest.

4. Failure Modes & Operational Reality#

4.1 Cardinality Explosion#

t=0:       Tenant deploys code adding tag request_path (unnormalized, includes IDs)
t=+2min:   tenant.new_series_rate 3K/min -> 420K/min
t=+2min:   Intake limiter: new series beyond 50K/min dropped; existing accepted
t=+3min:   Event to tenant: metric http.requests, tag request_path, ~3.1M distinct values/10min
t=+3min:   Shard memory +4% (bounded by limit) instead of OOM
t=+40min:  Tenant ships path normalization; new-series rate back to baseline

• Detection: intake.new_series_rate{tenant}, intake.series_dropped{tenant,metric}, per-metric HLL estimates.
• Blast radius: Without limits, every tenant on the shard; with limits, only the offending tenant's new series.
• Mitigation: Drop new series, notify, offer tag allowlist.
• Prevention: Default allowlists for known-dangerous tag names (user_id, request_id, trace_id); agent-side normalization of URL paths.
• Owner: Platform owns limits; tenant owns instrumentation.

4.2 Alert Evaluator Lag During a Regional Incident#

t=0:       Cloud region partial outage; 30% of tenants' error metrics spike
t=+1min:   Ingest +40% (retries, error metrics, autoscaled hosts)
t=+2min:   Kafka consumer lag for evaluators: 5s -> 3min on 12 partitions
t=+3min:   alerting.eval_lag_p99 = 190s > 120s SLO -> page platform on-call
t=+5min:   Evaluator autoscale adds 40% capacity; partitions rebalanced
t=+9min:   Lag back under 30s; pages delivered late by up to 4 min

• Detection: alerting.eval_lag_p99, kafka.consumer_lag{group=evaluators}, notifier.delivery_latency_p99.
• Blast radius: Delayed pages for tenants on lagging partitions — during their incident.
• Mitigation: Pre-provisioned evaluator headroom (≥ 2× steady state), prioritize partitions by alert density, shed dashboard queries if they share any resource.
• Prevention: Evaluators never share compute with query serving; capacity tests at 3× ingest.
• Owner: Alerting team.

4.3 The Silent No-Data Failure#

The most dangerous failure: a pipeline bug drops one metric family, and alerts on it go quiet because their no-data policy is "resolve."

t=0:       Agent release changes metric name nginx.requests -> nginx.http.requests
t=+10min:  Monitors on nginx.requests return no data; policy = resolve (tenant default)
t=+10min:  Dashboards show a flat line; nobody looks
t=+3days:  Customer's nginx error spike goes unpaged; outage lasts 47 min

• Detection: Platform-side: alerting.monitors_no_data{tenant} jump after a release; per-metric-name volume drop > 90% after an agent version change.
• Blast radius: Every monitor on the renamed metric across all tenants on that agent version.
• Mitigation: Emit both names for 2 releases; notify tenants with affected monitors.
• Prevention: Metric names are a public API with deprecation windows; agent releases canaried with a "metric-name diff" check; monitor creation defaults to "notify on no data" for infrastructure metrics.
• Owner: Agent team for naming stability; alerting team for no-data defaults.

🎯 Staff Move: "Metric names are an API. Renaming one silently disables every alert built on it, so they get deprecation windows like any other interface."

4.4 Query of Death#

• Symptom: A dashboard widget sum:*{*} by {container_id} over 30 days fans out to 40M series; query nodes OOM; dashboards for the cell slow to 30 s.
• Detection: query.series_scanned_p99, query.memory_peak, query node OOM restarts.
• Mitigation: Kill the query; per-query series cap (e.g., 500K) and memory cap; cost estimate before execution using the tag index.
• Prevention: Planner rejects or auto-downsamples queries over cost thresholds; per-tenant query concurrency slots.
• Owner: Query team.

4.5 Monitoring Is Down and Nobody Knows#

• Symptom: Intake load balancer misconfiguration drops 100% of traffic for one cell. Dashboards show gaps; alerts go to no-data or resolve.
• Detection: The independent watchdog (different cloud account, different pager provider) sends synthetic points every 10 s and expects a synthetic alert within 2 min. If not received, it pages the platform team directly.
• Blast radius: One cell's tenants — and they don't know unless we tell them.
• Mitigation: Roll back; publish a status page and notify tenants that alerts in the window may be missing.
• Owner: Platform SRE; the watchdog is owned by a separate team or rotation.

4.6 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Deep compression knowledge ≠ monitoring design. XOR encoding is solved; cardinality isn't.
• PromQL fluency ≠ platform thinking. Writing queries is different from bounding their cost.
• "We use Kafka" ≠ decoupling. The point is two independent consumers with different SLOs.
• Big ingest numbers ≠ scale thinking. Series churn from autoscaling is the scale problem.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Compression bit layouts — name Gorilla, move on.
• Query language grammar.
• Dashboard rendering and UI.
• Log and trace storage (different problem; mention correlation only).

6.4 Follow-Up Questions to Expect#

1. "How do you handle counters that reset when a process restarts?"
2. "How would you evaluate 10 million monitors every minute?"
3. "What does the query planner do for a 90-day range?"
4. "How do you rebalance a hot Kafka partition that holds one huge tenant?"
5. "How do you bill fairly — by host, by series, or by points?"
6. "How would you implement SLO burn-rate alerts?"
7. "A customer says an alert didn't fire. How do you prove what happened?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design Datadog."

Drill 2: Size It in Series#

Prompt: "How much memory does the ingest tier need?"

Drill 3: The Tag Explosion#

Prompt: "A customer adds customer_id (2M values) as a tag on their busiest metric."

Drill 4: The Alert Path#

Prompt: "Design the alert evaluation service for 5 million monitors."

Drill 5: Late and Missing Data#

Prompt: "An agent's network is flaky; its points arrive 3 minutes late. What do the alerts do?"

Drill 6: Multi-Tenant Noisy Neighbor#

Prompt: "Your largest customer is 8% of all series and just tripled."

Drill 7: Percentiles Over a Year#

Prompt: "Show me p99 latency per month for the last 12 months."

Drill 8: Build vs Buy#

Prompt: "We're a 400-engineer company paying a vendor $3M/year. Should we build our own?"

Drill 9: Rolling Out a New Limit Without Breaking Alerts#

Prompt: "We need to enforce a 1M-series cap on all tenants. 3% of tenants are above it."

Drill 10: Multi-Region#

Prompt: "Customers in the EU need data to stay in the EU, and a US region outage shouldn't stop EU alerts."

8. Deep Dive Scenarios#

Deep Dive 1: Peak Traffic — A Major Cloud Outage Hits 30% of Tenants#

Context: A large cloud provider has a regional outage. 30% of your tenants are affected at once. Ingest rises 60% (error metrics, retries, autoscaled replacement hosts creating new series), dashboard queries rise 12×, and alert notifications rise 40×. Your status page gets a complaint that alerts arrived 6 minutes late.

Questions to Surface First:

• Which path is lagging: intake, Kafka consumers for evaluators, evaluation compute, or notification delivery?
• Are dashboards and evaluators sharing any resource?
• Is new-series churn from replacement hosts hitting limits and dropping legitimate series?

Typical L5 Approach: Autoscales everything. Autoscaling takes 5–10 minutes to catch up; meanwhile, alerts lag and dashboards time out.

Staff Approach: Applies the shed order: throttles ad-hoc and long-range queries, lowers dashboard auto-refresh to 60 s, prioritizes evaluator consumption over storage writers on the log (storage can catch up later from Kafka), and raises new-series limits temporarily for affected tenants whose churn is from autoscaling. Checks notifier provider rate limits — 40× notifications may hit a paging provider's API limit, so batching and failover matter.

Principal Approach: Treats correlated customer incidents as the design point for capacity, not an anomaly. Sets headroom targets based on the worst historical correlated event (e.g., ≥ 3× ingest and ≥ 10× queries for alerting-critical paths), runs a yearly game day simulating a major cloud-region outage across all cells, and negotiates higher burst limits with paging providers in contract.

t=0:       Cloud region degraded; affected tenants' error metrics spike
t=+2min:   Queries 12x; query tier CPU 90%
t=+3min:   Notifications 40x; paging provider returns 429 on 20% of sends
t=+4min:   eval_lag_p99 150s on 18 partitions
t=+6min:   Shed order applied; evaluator lag falls to 40s
t=+8min:   Secondary paging provider engaged; delivery p99 back under 30s

Key Takeaway: "Your peak is your customers' correlated worst day. Protect the page, shed the dashboard."

What clears the Staff bar:

• Applies a pre-defined shed order
• Checks the notifier and external providers, not just internal compute
• Distinguishes autoscaling churn from abusive cardinality

Deep Dive 2: The Silent Failure — Alerts Stopped for One Metric Type#

Context: A customer reports an outage that should have paged. Their monitor on rate(errors) stayed OK for 50 minutes while errors spiked. Your evaluator logs show no errors. The monitor evaluated every minute.

Questions to Surface First:

• What did the evaluator actually compute each minute? Can we replay it?
• Did the underlying data arrive and get stored?
• Was anything deployed recently in the counter-handling path?

Typical L5 Approach: Checks the evaluator is healthy (it is), checks the data is stored (it is), and concludes the customer's threshold was wrong.

Staff Approach: Replays evaluation from the Kafka log for that window with the same code version and finds the rate computation returned 0: a deploy changed counter-reset detection, so every time the customer's process restarted (it was crash-looping), the new counter value was treated as a reset and the delta dropped. The crash loop was the incident — so the metric designed to catch it was suppressed by it. Fix the logic; add golden-test fixtures for counter resets; add an "evaluation audit" that samples monitors and recomputes them independently.

Principal Approach: Establishes that alert correctness is testable and must be tested continuously: synthetic monitors covering every aggregation type (gauge, counter with resets, distribution, ratio, no-data) run in production against synthetic series with known answers, and any mismatch pages the alerting team. Makes "alert evaluation replay" a customer-facing feature so disputes are resolved with evidence.

Key Takeaway: "An evaluator that computes the wrong answer looks exactly like a healthy one. Test alert semantics continuously with known answers."

What clears the Staff bar:

• Replays from the log rather than trusting logs of the evaluator
• Sees the irony — the incident suppressed its own signal — and designs against it
• Proactive customer disclosure

Deep Dive 3: Large-Customer Onboarding — 200M Series#

Context: An enterprise wants to migrate from a self-hosted Prometheus fleet: 200M active series, 1.5M series/hour churn from Kubernetes, 40K alert rules, and 5 years of data they want imported.

Questions to Surface First:

• What fraction of series are queried or alerted on?
• Which labels drive churn (pod, container_id, instance)?
• Do they really need 5 years of raw data, or rollups?

Typical L5 Approach: Provisions capacity for 200M series in the shared cells and bulk-imports the history.

Staff Approach: Runs a cardinality analysis on their current data first: often ~40–60% of series are never queried in 30 days, and pod-level labels drive most churn. Proposes intake-side aggregation dropping pod for most metrics (keeping deployment and namespace), reducing to ~70–90M series. Places them in a dedicated cell sized for 2× that. Imports history as 1 h rollups for > 15 months, 1 m for the last 15 months. Translates alert rules with automated tooling and runs both systems in parallel for 2–4 weeks comparing alert firings.

Principal Approach: Uses this as the template for enterprise migrations: a migration kit (rule translator, parallel-run comparator, cardinality analyzer), a pricing model that rewards series reduction, and an account plan with a named platform engineer through cutover.

Key Takeaway: "Onboard the cardinality, not just the capacity. Half of most series are never read."

What clears the Staff bar:

• Reduces before provisioning
• Parallel-run alert parity as the cutover gate
• Rollup-based history import

Deep Dive 4: Post-Mortem — The Monitoring Platform Was Down for 38 Minutes#

Context: A configuration push to the intake fleet contained a malformed tenant-limit file. Intake nodes rejected all traffic for three cells. Customers' dashboards showed gaps; many monitors went to NO_DATA or resolved. Your own internal alerting — which runs on the same platform — also went silent, so the team learned from a customer tweet after 22 minutes.

Questions to Surface First:

• Why did a config push reach three cells at once?
• Why didn't intake fail static on bad config?
• Why was our detection dependent on the thing that failed?

Typical L5 Approach: Adds validation to the config file and rolls back.

Staff Approach: Three fixes. Config pushes roll out cell by cell with automated health gates (ingest rate per cell must stay within 10% of baseline for 10 minutes before the next cell). Intake fails static: a config that fails to parse keeps the last-known-good. And an external watchdog — separate cloud account, separate paging provider — sends synthetic metrics end-to-end and expects a synthetic alert every minute.

Principal Approach: Codifies "monitoring must not depend on what it monitors" as a platform rule: the platform's own alerting runs on an independent minimal stack; every global config channel is audited for blast radius; and customers get a documented status API so their own tooling can detect our outage.

t=0:       Config push to all intake nodes in 3 cells
t=+30s:    Intake rejects all payloads (limit file parse error)
t=+1min:   Agents buffer to disk; ingest rate for 3 cells = 0
t=+2min:   Our internal monitors (on the same cells) go NO_DATA, policy resolve
t=+22min:  Customer tweet; on-call investigates
t=+31min:  Config rolled back
t=+38min:  Agent buffers drained; data gaps partially backfilled

Key Takeaway: "The monitoring system can't detect its own death. Something outside it must."

What clears the Staff bar:

• Treats config as code with staged rollout
• Fail-static for control-plane inputs
• Independent detection path

Deep Dive 5: Multi-Region Expansion — Launching an EU Region#

Context: Legal requires EU customer telemetry to stay in the EU. Some global customers want one dashboard across US and EU. The EU region launches with 5% of current capacity.

Questions to Surface First:

• Which data counts as customer telemetry — tags can contain personal data (hostnames, user emails in tags)?
• Is cross-region query acceptable if data doesn't leave the region at rest?
• How do we handle a tenant with hosts in both regions?

Typical L5 Approach: Deploys a copy of the stack in the EU and routes EU tenants there.

Staff Approach: Independent regional stacks with tenant home regions; agents configured with the regional intake. For tenants spanning regions, each region stores its own hosts' data; a federated query layer merges results at query time without persisting cross-region copies. Alert rules run in the region of their data; cross-region rules evaluate on the federation layer with degraded semantics (partial data flagged) — and the default recommendation is region-local monitors. The EU watchdog runs from outside the EU region.

Principal Approach: Treats residency as a repeatable region-launch capability: a checklist, infrastructure-as-code for a full regional stack, legal review of which fields are telemetry vs account data, and a pricing tier for residency. Plans the capacity ramp so a small new region still has cells, headroom, and on-call coverage.

Key Takeaway: "Regions are independent failure and legal domains. Federate reads; never federate alerts by default."

What clears the Staff bar:

• Region-local alerting
• Query-time federation without persisted copies
• Cross-watchdogs between regions

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Explain why active series — not points or bytes — are the unit of cost, memory, and failure
• Design intake-side cardinality limits that protect existing series and name the culprit tag
• Separate alert evaluation from query serving with two consumers off a durable log
• Specify evaluation delay, no-data policy, and hysteresis for alert rules
• Design tiered retention with rollups that stay correct (sum/count/min/max, sketches)
• Isolate tenants with cells, shuffle sharding, and query cost limits
• Detect a monitoring outage from outside the monitoring system
• Price the platform per series at three scales and name its one-way doors

The Bar for This Question#

Mid-level (L4): Agents write to a TSDB; dashboards query it; an alert job polls. Sizing by data volume. Single-tenant thinking.

Senior (L5): Adds Kafka, sharded storage, retention, replication, and a separate alert service that queries storage. Solid — but sizes by points, has no cardinality limits, shares query capacity between dashboards and alerts, averages percentiles in rollups, and relies on the platform to detect its own outages.

Staff+ (L6): Sizes and limits by series; isolates the alert path; defines late and no-data semantics; stores sketches for percentiles; uses cells and query cost limits; applies a shed order under load; runs an independent watchdog; names owners for agent naming stability, alert semantics, and tenant instrumentation. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Most Metrics Are Never Read"#

The Staff position: Show "unqueried series" per tenant and per team; default to aggregation that drops ephemeral labels unless requested.

Why this matters in interviews: It turns "scale the TSDB" into "reduce what we store" — a more senior move.

10.2 "Per-User Metrics Are a Category Error"#

The Staff position: Unbounded dimensions go to traces/logs; metrics carry bounded dimensions only.

Why this matters in interviews: It shows you choose the data type by question, not by habit.

10.3 "Threshold Alerts Are Mostly Noise — Alert on SLO Burn"#

The Staff position: Page on SLO burn rate; route resource thresholds to tickets.

Why this matters in interviews: It connects platform design to on-call health.

10.4 "Monitoring Should Be Less Available Than Alerting"#

The Staff position: Separate SLOs: alerting 99.99% with p99 < 2 min; dashboards 99.9% with a shed policy.

Why this matters in interviews: It shows you design explicit degradation rather than uniform availability.

10.5 "Build Your Own Only at Planet Scale"#

The Staff position: Buy or run open source; build the governance layer (limits, attribution, standards) yourself.

Why this matters in interviews: Candidates who propose writing a TSDB for a mid-size company signal poor TCO judgment.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer builds a monitoring platform that stays up during incidents. A Principal engineer notices that observability spend is growing faster than infrastructure spend — often 10–30% of the infra bill — because every team instruments independently, cardinality is invisible to the engineer who adds a tag, and alert noise is nobody's metric. The technical platform is solvable; the org's instrumentation economy isn't, until someone makes cost and alert quality visible and owned. L7 decides what to standardize (instrumentation libraries, naming, tag rules, SLO-based paging), what to leave to teams (which metrics matter), and whether to build, buy, or run hybrid.

The Org-Level Fault Line#

Central observability platform with enforced standards vs team-chosen tools and instrumentation.

The L7 default: Paved road built on OpenTelemetry instrumentation (vendor-neutral), a single agent, naming and tag standards enforced in CI, per-team series budgets with chargeback, and SLO burn-rate paging as the default alert type.

🧭 Principal Move: "The cheapest series is the one never emitted. I'd put the cardinality estimate in the pull request that adds the tag, with the monthly cost next to it."

Cost Model#

Assumptions: fully loaded cost per million active series ~$1.5–3K/month (RAM-heavy ingest heads, index, evaluators, object storage for blocks and rollups, Kafka); 10 s resolution; 15-day raw and 15-month rollup retention; engineer ~$25K/month loaded.

Comparison worth saying: a mid-size company paying a vendor for ~10M custom series at list prices typically pays more than the self-hosted infra above — but the self-hosted column's headcount usually closes the gap until spend reaches several million dollars a year.

The 3-Year Evolution Path#

Triggers: Year 1 when a single tenant causes a cross-tenant incident; Year 2 when alert noise or cost reaches leadership; Year 3 when a residency-driven customer or region launch requires it.

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Metrics & Alerting Standard v1

Scope: All services emitting operational metrics or defining pages.

MUST:

• Instrument with the platform OpenTelemetry SDK; metric names follow <domain>.<component>.<measure> with units in the name.
• Tags have bounded value sets; user_id, request_id, trace_id, raw URLs, and email addresses are prohibited as metric tags (enforced by CI lint and intake).
• Latency is recorded as a distribution, not as precomputed percentiles.
• Every paging monitor has an owning team, a runbook link, an explicit no-data policy, and — for user-facing services — is an SLO burn-rate alert.
• Teams stay within their series budget or file for an increase with cost acknowledgment.

SHOULD:

• Resource-threshold alerts route to tickets, not pages.
• Review unqueried series quarterly and remove them.

Exceptions: Filed with the observability platform team; approved by platform lead and the requesting team's director; expire in 6 months.

Success metrics: Actionable-page ratio ≥ 70%; pages per on-call shift ≤ 2 median; observability cost growth ≤ infrastructure cost growth; 100% of paging monitors owned.

What I'd Tell the VP#

Our monitoring bill is growing faster than our infrastructure because engineers can add expensive measurements with one line of code and never see the cost. At the same time, our on-call engineers get paged for things that don't affect customers, which makes them slower to respond when something does. I'm proposing a standard toolkit, a budget per team with visible costs, and paging based on customer-facing reliability targets instead of server thresholds. Expect a 25–40% reduction in observability spend within two quarters and fewer, more meaningful pages. It needs about four engineers for two quarters, and it keeps us vendor-neutral so we can renegotiate or switch providers later.

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "Per-tenant limits protect the platform" — correct, but doesn't change the behavior of the engineers generating cardinality.
• "Alert evaluation is isolated and fast" — but nothing about whether the alerts are worth sending.
• "We'll use Prometheus and Thanos" — a stack choice, not a strategy for cost, standards, and ownership.

Appendices

Appendix A: Storage Mechanics in Depth#

A.1 Gorilla-Style Compression#

Timestamps: store t0, then delta d1 = t1 - t0, then delta-of-delta dd = (tn - tn-1) - (tn-1 - tn-2)
  dd == 0            -> 1 bit '0'         (regular 10s intervals: almost always)
  dd in [-63, 64]    -> '10' + 7 bits
  larger             -> wider buckets
Values: XOR with previous value
  xor == 0           -> 1 bit '0'         (unchanged gauge)
  else               -> leading/trailing zero counts + meaningful bits
Average ~1.37 bytes/point on production data (Facebook, 2015)

A.2 Head, Blocks, and Compaction#

• Head: per-series open chunk (~120 points), write-ahead log for crash recovery, in-memory inverted index.
• Block cut: every 2 h, flush chunks and the index as an immutable block to object storage.
• Compaction: merge 2 h blocks into larger blocks (e.g., 24 h); produce 1 m rollups; drop expired raw data at 15 days.
• Query: planner selects head for the last 2 h, raw blocks for ≤ 15 days, rollups beyond.

A.3 Inverted Index#

postings["service=checkout"] = sorted [series_id...]
postings["region=us-east"]   = sorted [series_id...]
query {service:checkout, region:us-east} = intersect(postings...)

Posting lists are compressed (delta + varint or roaring bitmaps). Regex/wildcard matchers expand against the tag-value dictionary first — the expansion size is a query-cost input.

A.4 Mergeable Sketches#

DDSketch buckets values logarithmically so relative error is bounded (e.g., 1%); two sketches merge by adding bucket counts. That makes per-host, per-10 s sketches mergeable into per-service, per-month sketches without error growth — the property percentile rollups need.

Appendix B: Series Identity and Data Model#

Appendix C: Coordination Mechanisms#

Quick comparison: Evaluating alerts in stream processors (Flink-style) gives the freshest results and natural watermarks; evaluating against a recent-data replica is simpler for complex queries. Most platforms use both, split by rule complexity.

Appendix D: Agent and API Contract#

Appendix E: Observability of the Observability Platform#

E.1 Core Metrics#

Intake:     intake.accepted_points{cell}, intake.series_dropped{tenant}, intake.new_series_rate{tenant}
Log:        kafka.consumer_lag{group}, kafka.under_replicated_partitions
Alerting:   alerting.eval_lag_p99, alerting.monitors_no_data, alerting.synthetic_mismatch_total
Notify:     notifier.delivery_latency_p99, notifier.delivery_failures{provider}
Storage:    ingest.head_series, ingest.memory_used_pct, compaction.backlog
Query:      query.latency_p95, query.series_scanned_p99, query.shed_total
Watchdog:   watchdog.synthetic_alert_latency (from outside)

E.2 Critical Alerts#

E.3 Control Plane vs Data Plane#

Control plane: monitor CRUD, tenant limits, cell routing, config. Data plane: intake, log, evaluation, notification, storage. The data plane runs on last-known-good control-plane state; control-plane changes roll out cell by cell with health gates (Deep Dive 4).

E.4 Debugging "My Alert Didn't Fire"#

1. Replay the monitor's evaluation from the log for the window.
2. Check data arrival: were points accepted, late, or dropped by limits?
3. Check semantics: evaluation delay, no-data policy, for duration, recovery threshold.
4. Check notification: state transition recorded? Delivery attempts and provider responses?

Appendix F: Scale Evolution#

F.1 What Works at Each Scale#

F.2 Multi-Region Path#

Independent regional stacks; tenant home region; query-time federation; region-local alerts; watchdogs placed outside the region they watch (Drill 10, Deep Dive 5).

F.3 What You Don't Build on Day One#

• Anomaly-detection alerting (start with thresholds and burn rates)
• Global federated query
• 1-second resolution
• Custom TSDB engine
• Dedicated cells (until a tenant reaches a meaningful share of a cell)

Appendix G: Multi-Tenancy, Fairness, and Cost#

🧭 Principal Insight: "Charge for what costs us: series and retention. If the pricing unit doesn't match the cost driver, customers will optimize for the invoice and we'll absorb the difference."

Related reading: Stream Processing, Message Queues, Notification Systems, Circuit Breakers, Degraded Mode, Data Pipelines, and Capacity Planning.