Design with Time Series DBs — Staff-Level Technology Guide

Why This Matters#

A time series database is not a database choice. It is a cardinality budget with a storage engine attached. Every TSDB — Prometheus, InfluxDB, TimescaleDB, M3, VictoriaMetrics — is fast at the same thing: appending timestamped numbers to a known series and reading a contiguous time range back. Every one of them dies the same way: somebody adds a label with a million distinct values, the in-memory index explodes, and the monitoring system goes dark exactly when the incident it was supposed to catch begins.

That is why this technology keeps showing up in Staff loops. "Design a metrics platform", "Design ad click aggregation", "Design an IoT telemetry pipeline", and "How do you monitor this system?" are all TSDB questions in disguise. The L5 candidate names Prometheus and draws a Grafana box. The L6 candidate says "active series is my capacity unit, not bytes; I'll cap cardinality per team at 500K series and reject user_id as a label." The L7 candidate asks who pays for the series — and whether observability should be a chargeback platform or a free-for-all that bankrupts itself in year two.

The gap between levels is not knowing what delta-of-delta encoding is. It is knowing that the write path is almost never the bottleneck — the index is, that percentiles cannot be averaged, that retention is a product decision with a dollar figure, and that the monitoring system must fail independently of the thing it monitors.

The L5 → L6 → L7 Contrast#

The 60-Second Pitch#

"For operational metrics I'd use a Prometheus-compatible TSDB — Prometheus at the edge for scraping and short-term alerting, remote-writing into a horizontally scalable long-term store like Mimir, Thanos, or VictoriaMetrics. The data is append-only, time-ordered, and numeric, so Gorilla-style compression gets us to under 2 bytes per sample — about 8–12× smaller than raw. The thing I'll actually design around is cardinality: I'm budgeting active series, not disk, and I'll enforce label limits at ingest. Retention is tiered — 15 days raw, 90 days at 5-minute rollups, 2 years hourly. If the question is business analytics with joins and ad-hoc slicing by customer, I'd switch to TimescaleDB or ClickHouse instead — a TSDB is the wrong tool for high-cardinality questions."

That pitch hits the four things interviewers listen for: a named engine with a reason, a compression number, the cardinality constraint, and the explicit boundary of when not to use it.

The Three Intents#

TSDB questions hide three different workloads. They lead to incompatible designs.

🎯 Staff Move: "I'll assume operational monitoring first — that's where freshness, cardinality, and 'the monitor must outlive the outage' collide. If this is really billing-grade usage metering, I'd move it off the TSDB entirely onto an exactly-once pipeline, because a TSDB is allowed to drop samples and a bill is not."

The Staff Positions#

Architecture & Internals#

Only four internals change design decisions: the compression scheme, the head block + WAL, the inverted index, and block compaction. Everything else is implementation detail.

The Data Model — Series, Labels, Samples#

A series is identified by a metric name plus a sorted set of label key-value pairs: http_requests_total{service="checkout", method="POST", status="500"}. A sample is a (timestamp_ms int64, value float64) pair. Raw, that is 16 bytes. The storage engine's whole job is to make it ~1.5.

Gorilla Compression — Why Samples Cost ~1.4 Bytes#

Facebook's 2015 Gorilla paper introduced the encoding that nearly every modern TSDB copies. Two observations drive it:

1. Timestamps are regular. Scrapes arrive every 15s, so the delta between timestamps is ~15000ms, and the delta of the delta is almost always 0. Store delta-of-delta with a variable-length prefix: a 0 bit when it is zero. That is 1 bit per timestamp in the common case.
2. Consecutive values are similar. XOR a float with the previous float; most leading and trailing bits are zero. Store only the meaningful middle bits. An unchanged value costs 1 bit.

timestamps:  1000, 1015, 1030, 1045, 1061
deltas:            15,   15,   15,   16
delta-of-delta:          0,    0,    1      -> '0' '0' '10'+bits
values:      42.0, 42.0, 42.5, 42.5
xor w/ prev:       0,    X,    0            -> '0' 'meaningful bits' '0'

Gorilla paper result: ~1.37 bytes/sample average vs 16 raw (~12x)
Prometheus in practice: ~1.3-2 bytes/sample on disk

Design consequences of this encoding:

• Counters compress better than gauges. A monotonically increasing counter with a stable rate has near-constant deltas.
• Jittery scrape intervals cost bytes. Irregular timestamps break the 1-bit path.
• High-precision random floats compress badly (~8+ bytes). Don't push sensor noise at 15 decimal places.
• Chunks are append-only, typically ~120 samples each. You cannot cheaply update a sample in the middle — updates and deletes are expensive in every TSDB.

The Write Path — Head Block, WAL, Blocks#

1. Append to WAL for crash recovery. Replaying a large WAL is why a Prometheus with 10M series can take 5–20 minutes to restart — an operational fact that belongs in your failure-mode discussion.
2. Append to the open chunk of the series in memory. New series also update the in-memory inverted index (label value → sorted list of series IDs, called postings).
3. Every 2 hours the head is cut into an immutable on-disk block: chunks + index + tombstones.
4. Compaction merges 2h blocks into larger blocks (in Prometheus, up to 31 days or 10% of the retention window), which shrinks the index and speeds long-range queries.
5. Retention deletes whole blocks. There is no row-level delete in the hot path — deletes are tombstones applied at compaction.

🎯 Staff Insight: "The head block is the entire operational story. Memory scales with active series, restart time scales with WAL size, and series churn — pods coming and going with new label values — inflates the index even if the steady-state series count looks fine. I'd alert on prometheus_tsdb_head_series and on churn rate, not on disk."

The Read Path — Postings Intersection#

A PromQL selector like http_requests_total{service="checkout", status=~"5.."} resolves by:

1. Look up the postings list for __name__="http_requests_total", for service="checkout", and the union of postings for every status value matching 5...
2. Intersect the sorted lists → matching series IDs.
3. For each series, decode only the chunks overlapping the query time range.
4. Apply functions (rate, sum by, histogram_quantile) in the query engine.

Query cost is roughly (series matched) × (samples per series in range). A dashboard panel touching 50K series × 24h × 4 samples/min = 288M samples to decode — that is what kills query nodes, not the number of dashboards.

Pull vs Push#

Staff default: pull inside a cluster, push across boundaries. Prometheus scrapes locally and remote-writes (push) to the central store. That hybrid is what Mimir, Thanos Receive, and VictoriaMetrics are built for.

Data Modeling — "The Entire Game"#

In Cassandra the game is the partition key. In a TSDB the game is the label set, because the label set is the primary key, and every distinct combination is a new series with its own memory cost forever (well, until it goes stale).

Cardinality Is Multiplicative#

series(metric) = product over labels of (distinct values of that label)

http_request_duration_seconds_bucket
  service   : 40
  endpoint  : 25   (per service, avg)
  method    : 4
  status    : 6
  le        : 12   (histogram buckets)
  pod       : 30   (per service, avg)
= 40 x 25 x 4 x 6 x 12 x 30 = 8.64M series  <- from ONE metric

Drop pod (aggregate at recording rule):  288K series
Add user_id (50M users):                 game over

The rule is simple and non-negotiable: every label must have a bounded, known set of values, and the product must fit the budget.

🎯 Staff Move: "I'd normalize the route to its template — /orders/:id, not /orders/83712 — in the HTTP middleware, because one raw path label is enough to OOM the cluster. And I'd attach the trace ID as an exemplar, not a label, so an engineer can jump from a p99 spike to one slow trace without paying for 50M series."

Metric Types and What They Cost#

The single most common correctness bug in metrics design is averaging percentiles:

pod A: p99 = 50ms  (10,000 req)
pod B: p99 = 900ms (100 req)
avg(p99) = 475ms   <- meaningless
true fleet p99 ~= 55ms

Correct:
histogram_quantile(0.99,
  sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

Schema Design in InfluxDB and TimescaleDB#

InfluxDB (1.x/2.x) splits a point into tags (indexed, part of series identity) and fields (values, not indexed). The same cardinality rule applies to tags. The classic mistake is making device_id a field (then you can't filter on it efficiently) or making reading_id a tag (series explosion).

# line protocol: measurement,tags fields timestamp
cpu,host=web-17,region=us-east usage_user=12.4,usage_sys=3.1 1735689600000000000

TimescaleDB is PostgreSQL with hypertables: one logical table auto-partitioned into time-based chunks. Because it is relational, high-cardinality dimensions are just columns with a B-tree index — which is exactly why it wins the "analytics over time with joins" intent.

CREATE TABLE readings (
  time       TIMESTAMPTZ NOT NULL,
  device_id  BIGINT      NOT NULL,
  site_id    INT         NOT NULL,
  temp_c     DOUBLE PRECISION,
  humidity   DOUBLE PRECISION
);
SELECT create_hypertable('readings', 'time', chunk_time_interval => INTERVAL '1 day');

-- Columnar compression on chunks older than 7 days, segmented by device
ALTER TABLE readings SET (timescaledb.compress,
  timescaledb.compress_segmentby = 'device_id',
  timescaledb.compress_orderby   = 'time DESC');
SELECT add_compression_policy('readings', INTERVAL '7 days');

-- Continuous aggregate: incrementally maintained hourly rollup
CREATE MATERIALIZED VIEW readings_1h WITH (timescaledb.continuous) AS
SELECT time_bucket('1 hour', time) AS bucket, device_id,
       avg(temp_c) AS avg_temp, max(temp_c) AS max_temp, count(*) AS n
FROM readings GROUP BY bucket, device_id;
SELECT add_retention_policy('readings', INTERVAL '30 days');

Chunk sizing rule of thumb: size chunk_time_interval so the most recent chunk(s) across all hypertables fit in ~25% of RAM — the hot chunk's indexes must stay in memory or insert latency degrades.

Rollups — Store Aggregates, Not Averages#

A rollup must preserve the ability to re-aggregate. Store sum, count, min, max (and histogram buckets), never just avg — an average of averages is wrong the moment the underlying counts differ.

rollup_5m(series) = { sum, count, min, max, last }   per 5-minute window
avg over any window = sum(sum) / sum(count)          <- correct
avg(avg)                                             <- wrong unless counts equal

The Tunable Tradeoff — Resolution × Retention × Cardinality#

Every TSDB design is a point in a three-dimensional budget. Pick two; the third is the bill.

storage_bytes/day ~= active_series x (86,400 / scrape_interval_s) x bytes_per_sample

  1M series, 15s, 1.5 B/sample  ~= 1M x 5,760 x 1.5  ~= 8.6 GB/day  (raw)
 10M series, 15s                ~= 86 GB/day   -> 15d raw ~= 1.3 TB
 10M series, 60s                ~= 21.6 GB/day

memory (head)  ~= active_series x 3-8 KB   (Prometheus; VictoriaMetrics ~1 KB)
 10M series    ~= 30-80 GB RAM on one Prometheus -> shard or move to Mimir/VM

Tiered retention is the Staff default:

🎯 Staff Move: "I'd ask product whether anybody has ever looked at 15-second resolution data older than two weeks. In every org I've seen, the answer is no — so raw retention is 15 days and everything beyond is 5-minute rollups. That single decision is usually a 10× storage reduction, and I'd have the SRE lead sign off on it because they're the ones who'll miss the data in a post-mortem."

Anti-Patterns — What Kills Time Series DB Deployments#

1. Unbounded Labels (The Cardinality Bomb)#

A developer adds customer_id to a latency histogram. Series count goes from 400K to 90M in the hour after deploy. Head memory OOMs, Prometheus restarts, WAL replay takes 15 minutes, and during those 15 minutes every alert in that cluster is silent. This is the #1 TSDB outage pattern, industry-wide.

Guardrail: per-tenant and per-metric series limits at ingest (Mimir max_global_series_per_user, VictoriaMetrics -maxLabelsPerTimeseries and series limiters, Prometheus sample_limit per scrape job). Reject, don't absorb.

2. Using the TSDB as an Event Store#

Writing one series per request, per order, or per click — "we'll query it later." A TSDB is optimized for many samples per series, not many series with one sample. Events belong in logs, Kafka, or a columnar OLAP store.

3. Averaging Percentiles and Storing Summaries#

Summaries compute quantiles client-side and cannot be aggregated. Fleet-wide p99 dashboards built on avg(summary_quantile) are silently wrong because every pod gets equal weight regardless of traffic: one idle, slow canary pod can drag the "fleet p99" up by 10×, and a few fast, busy pods can hide a real regression. The SLO report is fiction, and nobody notices until a customer does.

4. Short-Lived Series Churn#

Kubernetes pods, CI jobs, and autoscaled workers each create new pod= label values. A cluster with 10K pods redeploying twice a day creates 20K new series per metric per day. Steady-state count looks fine; the index over the retention window does not. Watch prometheus_tsdb_head_series_created_total rate.

5. Monitoring Inside the Blast Radius#

Prometheus running in the same Kubernetes cluster, on the same nodes, behind the same DNS, alerting through the same network path as the service it watches. When the cluster fails, the pager is silent. Classic correlated failure.

6. Unbounded Dashboard Queries#

A "last 30 days" dashboard with sum by (pod) over 500K series, auto-refreshing every 10s, opened by 40 engineers during an incident. Query nodes OOM exactly when they are needed most. Fix: query limits (max_samples, max_fetched_series_per_query), recording rules for heavy panels, and results caching.

7. Metering Revenue from a TSDB#

TSDBs deduplicate HA pairs, drop out-of-window samples, and downsample. Billing from sum(increase(api_calls_total[30d])) means under-billing on every counter reset edge case and no audit trail. Billing reads from an exactly-once event pipeline.

The Technology Landscape — Head-to-Head Comparison#

Managed/vendor options — Datadog, Grafana Cloud, Amazon Managed Prometheus / Timestream, Google Cloud Monitoring — collapse ops to zero and typically price per active series, per custom metric, or per sample ingested. That pricing model is exactly aligned with the cardinality problem, which is why an unreviewed label change can show up as a five- or six-figure invoice line.

Patterns#

Pattern 1: Edge Prometheus + Central Long-Term Store (the Staff default)#

• Alerting stays local — a central-store outage or WAN partition does not silence pages.
• HA pairs scrape identically; the central store deduplicates by cluster + __replica__ labels.
• Object storage makes 13-month retention cheap (~$0.02/GB-month).
• When to use: more than one cluster, or retention beyond ~30 days, or more than ~5M series total.

Pattern 2: Federation / Recording-Rule Rollup#

A global Prometheus scrapes only pre-aggregated recording-rule series (job:http_requests:rate5m) from per-cluster Prometheus instances. Cheap and simple; loses raw drill-down. Use for small orgs (<5 clusters) that need a global SLO view without running Mimir.

Pattern 3: Push via Kafka for IoT and Events#

Devices → MQTT/HTTP gateway → Kafka (partitioned by device_id) → consumers batch-write to InfluxDB/TimescaleDB/VictoriaMetrics. Kafka absorbs bursts, allows replay after a TSDB outage, and fans out to the OLAP store. Use when producers are untrusted, bursty, or out-of-order. See Apache Kafka.

Pattern 4: Streaming Pre-Aggregation#

Instead of storing per-pod series and aggregating at query time, aggregate in the pipeline (M3 Aggregator, Flink, or vmagent stream aggregation) and store only the rollup. Cuts series 10–100×; loses the ability to drill into a single pod after the fact. Use for high-churn fleets where per-instance history is never queried. See Flink & Stream Processing.

Pattern 5: Dual-Store — TSDB for Health, OLAP for Questions#

Aggregates go to the TSDB for dashboards and alerts. Raw events with full dimensions (customer, SKU, experiment arm) go to ClickHouse/Druid/BigQuery. The TSDB answers "is it broken?"; the OLAP store answers "for whom?" Use whenever someone asks for per-customer latency.

Scaling#

Where Each Engine Hits the Wall#

Sharding Strategies#

• Functional sharding — one Prometheus per team or domain. Simple, clear ownership, but cross-team queries require a global layer.
• Hash sharding by series — hashmod relabeling splits targets across N Prometheus instances. Distributes load; every query fans out to N.
• Ingester sharding in Mimir/Cortex — distributors hash each series to a consistent-hash ring of ingesters, RF=3. Shuffle sharding assigns each tenant a subset of ingesters so one tenant's cardinality bomb cannot touch every ingester.

Ingest and Query Numbers#

Multi-Region#

Metrics are one of the few datasets where you should not replicate synchronously across regions. Each region runs its own ingest and alerting; a global query layer (Thanos Query, Mimir with federated querier, or Grafana with multiple data sources) fans out read-time. Cross-region replication is only for DR of long-term blocks, and object storage cross-region replication handles that asynchronously at ~$0.02/GB transfer.

🎯 Staff Insight: "If us-east is on fire, I want us-east's alerting to keep working even if the central store in us-west is unreachable, and I want us-west's alerting to not care at all. So alerting is region-local, storage is region-local, and only the dashboard query layer is global."

Failure Modes & Recovery#

1. Cardinality Explosion → Head OOM#

• Symptom: Prometheus/ingester RSS climbs vertically after a deploy; OOM-kill; restart loop.
• Root cause: A new label with unbounded values (user_id, raw URL path, error_message).
• Detection: prometheus_tsdb_head_series rate-of-change > 20% in 10 min; cortex_ingester_memory_series per tenant; scrape_series_added per job.
• Fix: Drop the label with metric_relabel_configs (labeldrop) or block the metric at the distributor; restart; accept the WAL replay gap.
• Prevention: Per-scrape sample_limit, per-tenant series limits, CI lint on instrumentation (reject labels matching *_id), a cardinality dashboard per team.

2. WAL Replay Blackout#

• Symptom: After a crash, Prometheus is "up" but serving no queries and evaluating no rules for 5–20 minutes.
• Root cause: Replaying a multi-GB WAL rebuilds the entire head index in memory.
• Detection: prometheus_tsdb_wal_replay_duration_seconds; readiness probe failing; dead-man's-switch alert fires.
• Fix: Wait, or fail over to the HA twin (which is why HA pairs exist).
• Prevention: Keep per-instance series under ~5M; enable memory-snapshot-on-shutdown (Prometheus 2.30+ feature flag) to cut replay; always run HA pairs.

3. Remote-Write Backlog#

• Symptom: Central dashboards show data 10+ minutes stale; edge Prometheus memory grows.
• Root cause: Central distributor throttling, network partition, or ingester slowness; remote-write queue shards max out.
• Detection: prometheus_remote_storage_highest_timestamp_in_seconds minus prometheus_remote_storage_queue_highest_sent_timestamp_seconds > 120s; prometheus_remote_storage_samples_failed_total.
• Fix: Scale distributors/ingesters; raise max_shards on the sender; drop non-critical metrics temporarily.
• Prevention: Alerting stays on the edge (so stale central data is an inconvenience, not an outage); WAL-based remote write retries for ~2h of backlog.

4. Query of Death#

• Symptom: Query nodes OOM or peg CPU; dashboards time out for everyone.
• Root cause: One query matching millions of series over weeks (sum by (pod) (rate(x[30d]))), often auto-refreshed.
• Detection: prometheus_engine_query_duration_seconds p99; cortex_query_frontend_queue_length; per-user query-sample counts.
• Fix: Kill the query; identify the dashboard via query logs.
• Prevention: query.max-samples (Prometheus default 50M), per-tenant max_fetched_series_per_query, query-frontend splitting by day with results cache, recording rules for expensive panels.

5. Silent Monitoring (Correlated Failure)#

• Symptom: Nothing. No alerts during an outage. Discovered from customer tickets.
• Root cause: Monitoring shares the failure domain — same cluster, DNS, network, or Alertmanager route to a paging provider that is itself down.
• Detection: Only external: a dead-man's-switch (vector(1) alert that must always fire) received by an external service that pages if it stops.
• Fix: Page from the external watchdog; move monitoring off the failing substrate.
• Prevention: Run monitoring on a separate cluster or account, cross-region meta-monitoring pairs (us-east watches us-west), quarterly game day that kills the monitoring stack.

Operational Reality Matrix#

When to Use vs. Alternatives#

When NOT to Use a TSDB#

• The question has an ID in it. "Latency for customer 8812" → OLAP or traces.
• Values must be exact and auditable. Billing, financial ledgers, compliance counts.
• Data is mostly updated, not appended. Current inventory, user profile state → OLTP.
• Events are sparse and unique. One sample per series is the worst case for every TSDB.
• You need joins across entities at query time — and you picked Prometheus. Use TimescaleDB or pre-join in the pipeline.

Operational Concerns#

What the On-Call Actually Does#

1. Watches cardinality, not disk. Per-tenant series count, series churn, top-10 metrics by series (topk(10, count by (__name__)({__name__=~".+"})) — run it on a replica, it's expensive).
2. Keeps the dead-man's switch green. An always-firing alert, received by an external service, paging if it goes silent for 5 minutes.
3. Manages limits. Most platform tickets are "my metrics are being dropped" → a tenant hit its series limit. The runbook is: show them the top offenders, then raise the limit only with a sign-off.
4. Watches compactor lag. In Mimir/Thanos, a stuck compactor means long-range queries touch thousands of uncompacted 2h blocks and slow down 10×. Alert when the oldest uncompacted block is > 24h old.
5. Handles upgrades as rolling restarts of HA pairs, one replica at a time, waiting for WAL replay to finish before the next.

Key Metrics & Alerts#

Governance That Actually Works#

• Shadow → warn → enforce for new limits: log would-be rejections for 2 weeks, warn owners, then enforce.
• Cardinality report per team, weekly, with trend. Most explosions are caught by the owning team once they can see it.
• Instrumentation library defaults — the shared HTTP middleware ships templated routes and a fixed histogram bucket set, so the right thing is the easy thing.

Interview Application — Staff-Level Plays#

Which Case Studies Use Time Series DBs#

Every System Design Question Has a TSDB Moment#

• URL shortener: "Click counts per link go to an OLAP store, not Prometheus — link ID is unbounded. Prometheus gets redirects_total{status} for health."
• Chat / messaging: "I'd track delivery latency as a histogram by region and client platform — 5 × 4 × 12 buckets = 240 series — and alert on the p99 burn rate against the SLO."
• Payment processing: "Authorization success rate by PSP and card network is my primary alert; the ledger itself never touches the TSDB."
• Job scheduler: "Queue depth and schedule lag per queue are gauges; per-job metrics would be unbounded, so job-level detail goes to logs with the job ID."

What Interviewers Probe#

Common Interview Mistakes#

L5 vs L6 vs L7 Responses#

The Staff TSDB Checklist#

1. State the series budget: "~2,000 services × ~500 series each ≈ 1M active series per region; 15s scrape ≈ 67K samples/sec."
2. Name the label policy: "Labels are bounded enums. Routes are templated. IDs go to exemplars and logs."
3. Choose the topology: "Prometheus HA pair per cluster for scrape and alerting; remote-write to a Mimir-class store for global queries and long retention."
4. Tier retention with a number: "15 days raw, 90 days at 5m, 2 years at 1h — roughly 10× cheaper than raw-forever."
5. Design the failure posture: "Alerting is local, the watchdog is external, and each tenant has series and query limits so one team can't blind the rest."
6. Draw the boundary: "Per-customer analytics and billing do not go here."

🎯 Staff Insight: Don't use a TSDB for anything with an unbounded identifier in the question, and don't bill from it. The strongest signal in a metrics interview is the moment you refuse a label.

Evaluation Rubric#

Strong hire signals

Lean no-hire signals

Common false positives

• Knowing Gorilla XOR bit layouts ≠ designing a metrics platform. Compression is solved; cardinality is not.
• Fluent PromQL ≠ capacity judgment. A beautiful query over 5M series is still a query of death.
• "We used Datadog" ≠ understanding the cost model. Ask what their bill was and why.

The Principal Lens#

Why L7 Sees This Problem Differently#

A Staff engineer sees a TSDB as a system to size and harden. A Principal sees it as the org's shared nervous system and its most elastic cost line. Every team emits metrics; almost no team pays for them. Observability spend commonly grows faster than the infrastructure it observes, because cardinality grows with services × instances × features while nobody's budget owns it. The L7 question is not "Mimir or VictoriaMetrics?" but "How do we make the cost of a label visible to the engineer who adds it, without making instrumentation so painful that teams stop measuring?"

The Org-Level Fault Line#

One central observability platform vs. every team runs its own stack.

The Principal default: central platform, per-team quotas with showback in year 1 and chargeback in year 2, and a paved-road instrumentation library that makes the cheap thing the default.

Cost Model#

Assumptions: 15s scrape, ~1.5 bytes/sample, RF=3 ingest, object storage $0.023/GB-month, compute at ~$0.04/vCPU-hr and ~$0.005/GB-RAM-hr, loaded engineer cost ~$250K/yr. Vendor figures assume typical list pricing per active series or custom metric — directional only.

The crossover where self-hosting beats vendor pricing typically lands between 1M and 5M active series — but only if the org can staff 2+ engineers who want to run it. Below that, buy.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

🧭 Principal Move: "The door I'd guard is the instrumentation SDK, not the database. If every service emits through OpenTelemetry or the Prometheus client with our shared middleware, I can swap backends in a quarter. If 400 services link a vendor agent, the vendor owns our roadmap."

The Standard I'd Write#

RFC-OBS-004: Metrics Instrumentation and Cardinality Standard

Scope: All services emitting metrics to the shared observability platform.

MUST
  1. Emit via the approved SDK (OpenTelemetry or Prometheus client) through the
     platform middleware; no direct vendor agents.
  2. Use only bounded labels. Labels named *_id, *_uuid, email, ip, path (raw),
     or free-text are rejected at ingest.
  3. Record latency as histograms using the platform bucket set; summaries are
     not accepted for fleet-level SLOs.
  4. Stay within the team's active-series quota (default 250K per team per region).
SHOULD
  5. Pre-aggregate per-instance metrics via recording rules when per-pod history
     is not needed beyond 15 days.
  6. Attach trace exemplars to latency histograms.

Exceptions: Filed with the observability platform team; approved by the platform
lead and the requesting team's director; time-boxed to 90 days; costed in the
request.

Rollout: shadow (log rejections) 4 weeks -> warn (dashboard + ticket) 4 weeks ->
enforce per region, canary region first.

Success metrics: series growth <= infra growth + 10%/yr; zero cardinality-caused
platform outages per quarter; p99 dashboard query < 2s; 100% of teams with a
visible showback line.

What I'd Tell the VP#

"Our monitoring cost is growing roughly twice as fast as our infrastructure because no team sees what their metrics cost. I'm proposing one shared platform with per-team quotas and a monthly cost report, moving to chargeback next year. This caps the bill at about $X per month at our projected scale, versus a vendor quote that grows with every new label. It also fixes the reliability risk: last quarter one team's change took alerting down for everyone, and quotas make that impossible. The ask is two engineers and a policy that directors sign."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll set per-tenant limits" — without saying who sets the quota, how exceptions work, or who pays for exceeding it.
• "Migrate to Mimir" — without a dual-write window, a dashboard migration plan, or a reason the org will fund it.
• "Use OpenTelemetry" — without recognizing it as a multi-year, every-team migration that needs an executive sponsor.

In the Wild#

Facebook — Gorilla#

Facebook built Gorilla as an in-memory write-through cache in front of its HBase-backed monitoring store, holding the most recent ~26 hours of data in memory. The 2015 paper introduced delta-of-delta timestamps and XOR float compression, averaging ~1.37 bytes per sample, and reported a 73× improvement in query latency. The key design choice was accepting that recent data matters most: almost all queries hit the last day, so optimize ruthlessly for that window and let older data live elsewhere.

Staff insight: The workload shape — recent data is hot, old data is rollups — justifies tiered retention. Quote the 1.37 bytes/sample number and then pivot immediately to cardinality, because compression is the solved half of the problem.

Uber — M3#

Uber built and open-sourced M3 (M3DB, M3 Coordinator, M3 Aggregator, M3 Query) after outgrowing earlier Graphite- and Cassandra-based metrics storage. M3 accepts Prometheus remote-write, stores data in a distributed TSDB with its own compression, and performs streaming aggregation so that downsampled series are computed at ingest rather than at query time.

Staff insight: At very large scale, you aggregate in the write path, not the read path. Say "streaming pre-aggregation" and name what you lose: per-instance drill-down after the raw retention window.

Netflix — Atlas#

Netflix built Atlas as an in-memory dimensional time series system for operational insight across its fleet, prioritizing fast queries on recent data and aggressive rollups for older data. It is designed around the reality that at Netflix scale the cost of keeping every dimension at full resolution is prohibitive, so dimensionality is managed deliberately.

Staff insight: Three companies, same conclusion — keep recent data hot and in memory, roll up aggressively, and treat dimensionality as a budget. That convergence is the argument to make out loud.

Practice Drill#

Prompt: "Our Prometheus-based monitoring for 3,000 services across 6 Kubernetes clusters keeps OOM-ing. Leadership wants 13 months of retention for capacity planning. Redesign it."

Quick Reference Card#

Sample raw size:        16 bytes (int64 ts + float64 value)
Gorilla compression:    ~1.37 bytes/sample (paper); Prometheus ~1.3-2 B
Head memory:            ~3-8 KB per active series (Prometheus); VM ~1 KB
Prometheus ceiling:     ~5-10M active series per instance
Head block:             2h in memory, WAL in 128MB segments
Samples/sec:            series / scrape_interval   (1M @15s = 67K/s)
Storage/day:            series x 5,760 x 1.5 B     (1M @15s ~= 8.6 GB/day)
Retention tiers:        raw 15s x 15d | 5m x 90d | 1h x 2y
Histograms:             8-15 buckets; aggregate buckets, never percentiles
Default query guard:    50M samples per query (Prometheus)
Remote-write lag alert: > 120s
Dead-man's switch:      always-firing alert, external receiver, page on silence

RED FLAGS
  - Label containing an ID, raw path, IP, or free text
  - avg() over a quantile
  - Monitoring in the same failure domain as the monitored system
  - Billing computed from a TSDB
  - "Prometheus scales horizontally"