Technologies referenced in this case study: Apache Kafka · Elasticsearch · OLAP Databases · Kubernetes · Apache Flink · Time-Series Databases

Related: Metrics & Alerting Platform · Distributed Tracing · Object Storage · Search Engine · Backpressure & Load Shedding · Multi-Tenancy · Batch and Stream Pipelines · ClickHouse vs Druid vs Pinot · Elasticsearch vs Postgres Full-Text · Security Basics

Reading Guide#

Organized for interview use first, reference second. Read front-to-back once, then return to the fault lines and incidents that match your weak spots.

Without per-tenant quotas and tiered storage:
t=0:        Payments service starts failing; retries log a 4KB stack trace per attempt.
t=+1min:    Payments log volume: 40 MB/s → 1.8 GB/s. Total cluster ingest: 2 GB/s → 3.8 GB/s.
t=+3min:    Indexing nodes saturate. Ingest lag for ALL services: 5s → 9 min.
t=+5min:    On-call searches for the payments error: results end 9 minutes ago.
t=+12min:   Index cluster rejects writes; agents buffer locally; some hosts' disks hit 95%.
t=+20min:   Incident resolved without logs. Post-mortem notes "logging was down".

With per-tenant quotas, agent buffering and a durable queue:
t=0:        Same failure. Payments volume spikes to 1.8 GB/s.
t=+10s:     Payments exceeds its burst quota. Pipeline samples repeated stack traces
            (first 100/min per signature kept in full, rest counted) and spills to object storage.
t=+1min:    Other services' ingest lag: unchanged at ~5s.
t=+2min:    On-call searches payments errors: sees the signature, the count, and full examples.
t=+3h:      Spilled raw logs queryable from object storage for the post-mortem, at lower speed.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What the Interviewer Is Scoring#

Log aggregation is not a search-engine question. Everyone can put Elasticsearch behind Kafka.

It is a cost and loss-policy question that tests:

• Whether you decide, in advance, whose logs are dropped, sampled or delayed when volume exceeds capacity
• Whether you match storage to query patterns — index the little that's searched often, scan the rest
• Whether you price retention per gigabyte per tier and attribute it to the teams producing the bytes
• Whether you keep secrets and personal data out of storage that is expensive to rewrite

The key insight: Log volume is driven by the people writing logs, not the people paying for them, and it spikes during incidents — exactly when logs are most valuable. A Senior design scales the cluster. A Staff design sets the policies that bound it: quotas per tenant, a loss order under backpressure, tiered retention with a price tag, and redaction at the edge. The platform's reliability is measured in whether the right logs are searchable during the worst hour, not in average ingest lag.

One Question, Three Levels#

Positions to Commit To#

Which Problem Are We Solving?#

Three intents produce three different systems. Name them, then commit.

🎯 Staff Move: "I'll design the operational debugging platform — high-volume service logs searched mostly during incidents. Audit logs get a separate lane with reserved capacity, no sampling and write-once storage; I'll show where they branch. If the real need is counting things, I'd push teams to emit metrics — counting error lines in a log store is the expensive way to build a dashboard."

Where the Design Splits#

How Real Companies Built It#

Why this section belongs here: Three well-documented systems made visibly different storage choices for logs. Naming them shows you see indexing as a cost decision, not a default.

Grafana Loki — Index Only Labels, Store Compressed Chunks in Object Storage#

Loki's documentation states that it does not index the contents of logs, only metadata as a set of labels for each log stream; log data is compressed and stored in chunks in an object store such as Amazon S3 or Google Cloud Storage, which keeps the index much smaller than other log tools (Loki overview). Its cardinality guidance warns that high-cardinality labels create many streams, a huge index and thousands of tiny chunks, and notes a default limit of 15 index labels (Loki cardinality).

Staff insight: This is the "query on read" end of the spectrum taken seriously: cheap ingest and storage, paid for with scan-heavy queries and strict label discipline. In an interview, say: "If I index only labels, cardinality limits become a tenant policy — one team putting request IDs in a label breaks the index for everyone."

Uber — From Elasticsearch to a Columnar Store for Schema-Agnostic Logs#

Uber's engineering blog describes moving its logging platform from the ELK stack to ClickHouse. With Elasticsearch, incompatible field types caused type-conflict errors that dropped logs, and the team ran 20+ clusters per region to limit the blast radius of heavy queries and mapping explosions. On ClickHouse, a single node ingested about 300K logs per second — roughly ten times a single Elasticsearch node in their setup — and the platform's hardware cost dropped by more than half while serving more traffic, with Kafka buffering ingestion (Uber blog).

Staff insight: Schema conflicts and mapping explosions are not edge cases at scale; they are the main failure mode of index-everything with dynamic fields. Say: "Dynamic field mapping is a multi-tenant hazard. Either I cap fields per tenant, or I store logs in a layout where a new field can't break someone else's index."

Datadog Husky — Stateless Writers, Object Storage and Separate Query Compute#

Datadog's engineering blog describes Husky, its third-generation event store for logs and other events, as an unbundled, distributed, schemaless, vectorized column store. Writers ingest from Kafka, buffer briefly and upload to blob storage; readers query individual files; compactors merge small files; and durability is pushed to FoundationDB and S3. The post says this separation made it cost-effective to retain data that is rarely queried but must stay immediately queryable, and enabled a new product, Flex Logs (Datadog blog).

Staff insight: Separating ingest, storage and query lets each scale with its own driver — bytes in, bytes kept, queries run — and turns retention into a pricing choice. Say: "Storage and query compute should scale independently, because logs are written a thousand times more than they're read."

Follow-Ups to Expect#

System Architecture Overview#

Reading the diagram: Agents on every node tail container output, apply a first redaction pass, and batch to an ingest gateway that tags the tenant and checks quotas. A durable queue absorbs bursts. Processors parse and enrich by stream schema, redact again with field-level rules, and apply overload control — sampling repeated messages by signature when a tenant exceeds quota. Every log line lands compressed in object storage; in-quota logs are also indexed in a short hot tier. Audit streams take a separate lane into write-once archive. The query frontend routes by time range: recent queries hit the hot index, older ones scan object storage. The metric that tells you the platform is healthy is ingest.lag_p99_s for in-quota tenants — it should stay under 30 seconds no matter which team is storming.

One-Minute Recap#

Numbers to Bring#

Interview Walkthrough

The most common mistake: Candidates spend 15 minutes on Elasticsearch shard counts, index templates and Kibana dashboards, then run out of time before the interviewer asks the questions that decide the level: "One team turns on debug logging and triples volume. What happens to everyone else?" and "What does 90 days cost?" Compress the pipeline to ~8 minutes and spend the rest on backpressure and loss policy, storage tiers, multi-tenancy, schema and redaction.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional scope in one breath:

"Every service on every host writes logs. We collect them, make them searchable within seconds by service, time, level and text, support live tail during incidents, keep them for a defined retention per stream, and give security an audit lane that's complete and tamper-evident."

Then the non-functional requirements, which is where the design lives:

"Four constraints drive everything. One: volume is bursty and correlated with failure — incidents multiply output 5–20× when search matters most. Two: logging must never take down the application. Three: cost scales with bytes, and producers don't see the bill unless we show it. Four: logs contain secrets and personal data whether we like it or not. I'll assume 3,000 services on 50,000 nodes, 2 GB/s average ingest, 10 GB/s during a bad incident, 7 days of fast search and 90 days of retained, slower search."

Then name the underspecified parts:

"I'd confirm: is this debugging, audit, or both? Who pays? What's the acceptable loss under overload? I'll assume debugging is primary with a separate audit lane, costs charged back per team, and debug-level logs sampled first under pressure."

🎯 Staff Move: Saying "incidents multiply log volume exactly when search matters most" reframes the problem from "store logs" to "stay useful during the worst hour" — that's the sentence that sets the level.

Phase 2: Core Entities & API (1–2 minutes)#

Name the nouns in 30 seconds:

• Tenant: a team or service owner — tenant_id, ingest_quota_bytes_s, burst_bytes, retention_policy, cost_center
• Stream: a labeled sequence — {tenant, service, env, cluster, level}; labels low-cardinality by policy
• Log record: ts, stream, body (raw), fields (parsed, optional), trace_id?, ingest_ts
• Chunk: compressed block of records for one stream and time range — chunk_id, tenant, t_min, t_max, bytes, tier
• Query: tenant, time_range, label_selector, filter (text/field), limit

Ingest API (agent → gateway):

POST /v1/ingest        Content-Encoding: gzip
Authorization: tenant token
{ streams: [ { labels: {service, env, level}, entries: [ [ts, line], … ] } ] }
→ 204 accepted
→ 429 quota exceeded, Retry-After: 2   (agent keeps buffering, backs off)
→ 413 batch too large

Query API:

GET /v1/query?selector={service="checkout",level="error"}&q="timeout"&from=-1h&limit=500
GET /v1/tail?selector={service="checkout"}           (live tail, server-sent stream)
POST /v1/retention   { stream_selector, hot_days, warm_days }

🎯 Staff Move: "429 with Retry-After is the backpressure contract between the platform and agents. The agent never drops on a 429; it buffers within its bounds and retries. Drops only happen when the agent's own bounded buffer is full — and each one is counted."

Phase 3: High-Level Architecture (≤5 minutes)#

Draw at most eight boxes:

Walk one log line in 90 seconds:

1. Checkout writes a JSON line to stdout. The node agent tails the container log, applies pattern redaction for tokens and card numbers, and adds it to a batch.
2. Every second or 1 MB, the agent gzips the batch and posts it to the gateway, which authenticates the tenant, checks the token-bucket quota and writes to the queue partitioned by tenant and stream.
3. A processor parses the line against checkout's stream schema, attaches Kubernetes metadata, runs field-level redaction, and checks the tenant's rate against quota. In quota: indexed into the hot tier and appended to a compressed chunk for object storage. Over quota: sampled by message signature, with full lines still written to object storage.
4. Within ~10 seconds the line is searchable in the hot tier; within ~5 minutes its chunk is flushed to object storage.
5. A query for the last hour goes to the hot index; a query for last month goes to object storage, filtered by labels and time, scanned in parallel.

🎯 Staff Move: Say out loud: "Every line lands in object storage, which is cheap and durable. The hot index is an accelerator for the recent window, not the system of record." You've now spent ~8 minutes.

Phase 4: Transition to Depth (1 minute)#

"That's the happy path, and it's the Senior-level design. What makes this hard is that volume is unbounded and spikes during incidents, cost scales with bytes nobody reads, and one tenant can hurt all the others. I'd like to go deep on backpressure and the loss order, index vs query-on-read and retention tiers, multi-tenant quotas and cardinality, schema handling, and PII redaction. Where would you like to start?"

If no preference: start with backpressure. It's the question that decides whether logs exist during the incident.

Phase 5: Deep Dives (25–30 minutes)#

For each: state the tradeoff → commit → quantify → name who pays.

Deep dive 1: Backpressure and the loss order (7–8 min)

"Four buffers, each bounded. The app's write to stdout never blocks on the network — the container runtime writes to a local file. The agent reads files with a 64 MB memory buffer that spills to a 1 GB disk buffer; when that's full, it stops reading new data and, if the file rotates away, those lines are lost and counted in agent.dropped_lines_total. The gateway returns 429 when a tenant exceeds quota; agents back off. The queue holds 48 hours, so an indexing outage of a day loses nothing. Overload control in the processor samples by signature for over-quota tenants — first 100 per signature per minute in full, the rest counted — but still writes everything compressed to object storage if the storage write path keeps up."

Quantify: "A 1 GB disk buffer at a typical node's 200 KB/s of logs is ~80 minutes of outage tolerance per host. 48 hours of queue at 2 GB/s is ~350 TB raw, ~35–70 TB compressed in the queue — real money, sized deliberately."

Who pays: "The storming tenant loses fidelity on its own repeated messages. Every other tenant pays nothing. Audit streams have reserved capacity and are never sampled."

Deep dive 2: Index vs query on read, and retention tiers (6–7 min)

"Query patterns decide this. Most searches are in the last 24 hours, filtered by service and level, then a text match. So the hot tier — 7 days — is indexed for interactive search in seconds. Everything is also in object storage as compressed, columnar chunks partitioned by tenant, stream and hour, with small per-chunk metadata — min/max time, labels, a bloom filter of tokens. Older queries prune by labels and time, skip chunks via bloom filters, and scan the rest in parallel. A 30-day query over one service is tens of seconds instead of sub-second; that's the explicit trade."

Quantify: "170 TB/day raw. Hot: 7 days indexed and replicated is roughly 170 × 7 × 2 × ~1 ≈ 2.4 PB of SSD-class storage if we indexed everything — which is why the hot tier holds only in-quota, non-sampled data and drops debug after 24 hours. Warm: 90 days × ~17 TB/day compressed ≈ 1.5 PB of object storage at ~$0.02/GB-month ≈ $30K/month."

Deep dive 3: Multi-tenant quotas and cardinality (5–6 min)

"Each tenant has an ingest quota in bytes per second with a burst bucket — say 2× quota for 10 minutes — enforced at the gateway and again in processing. Query limits per tenant: concurrent queries, bytes scanned per query, and a max time range for hot-tier queries. Cardinality limits: max labels per stream (around 10–15), max distinct values per label per day, max parsed fields per stream. Violations are rejected or folded into the raw body, never allowed to grow the shared index without bound."

Deep dive 4: Schema on write vs on read (4–5 min)

"Structured JSON is encouraged and parsed at ingest against a per-stream schema that the owning team registers. Known fields are typed and indexed in the hot tier. Unknown fields stay in the raw body and are searchable as text, never auto-added to a shared mapping. Type conflicts — status as a string in one service and an integer in another — can't collide because fields are namespaced per stream, and a value that fails to parse is kept raw, not dropped."

Deep dive 5: PII redaction and ownership (3–4 min)

"Two passes. The agent runs cheap pattern redaction — bearer tokens, API key formats, card numbers validated with a checksum to avoid false positives. The processor applies per-stream field rules: allowlisted fields kept, known-sensitive fields hashed or dropped. Seeded canary secrets in synthetic traffic verify coverage daily. When something slips through, the purge job rewrites the affected chunks — bounded because chunks are partitioned by tenant and hour. The platform team owns ingest health and quotas; product teams own their volume, schemas and what they log; security owns redaction rules and the audit lane."

Phase 6: Wrap-Up (2–3 minutes)#

"The core idea: log volume is driven by producers and spikes during incidents, so the platform is defined by its policies — a loss order under backpressure, an indexed hot window over cheap object storage, per-tenant quotas and cardinality limits, schemas that can't break each other, and redaction before persistence — with cost per gigabyte visible to the teams producing it."

The evolution closer:

"What I'd build later: log-to-metric extraction so teams stop querying logs for counts; trace-linked logs; per-team retention self-service with a budget; and federated queries across regions. What I'd not build: indexing every line for 90 days — I'd invest in reducing volume at the source instead."

🎯 Staff Move: End on who owns the bytes. "The platform owns keeping in-quota logs searchable within 30 seconds through any incident. Teams own their volume and see its cost. When a team's own storm gets sampled, that's the policy working, not the platform failing."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Logging is the interview where the users who generate the load are not the users who pay for it, and where demand spikes at the exact moment the system is most needed. Every engineer adds log lines; nobody removes them. Volume grows with services, traffic and incidents, and the cost line grows with it until a finance review asks why observability costs more than the database fleet.

A Senior design scales storage to match demand. A Staff design recognizes that demand is a policy problem: what gets kept, at what fidelity, for how long, at whose expense — and what gets dropped first when the system is overwhelmed. The architecture follows from those policies, and the policies need owners outside the platform team.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"It's the worst outage of the year. Log volume is 6× normal, one service is producing 80% of it, and the on-call is searching for a specific error in a different service. Walk me through what your platform is doing right now, whose logs are being dropped or sampled, and what that on-call sees."

A candidate who answers with the storming tenant hitting its burst quota, signature sampling applied to that tenant only, full raw lines still landing in object storage, queue depth rising but within its 48-hour bound, other tenants' ingest lag unchanged at seconds, query limits stopping a dozen engineers' 30-day searches from starving the hot tier, and the on-call finding their error in ten seconds has run a logging platform. A candidate who says "Kafka absorbs the spike" has drawn one.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Operational debugging → useful during the worst hour, affordable the rest of the time

• Constraint: bursty volume, recency-heavy queries, tolerance for sampling repeated debug lines
• Strategy: bounded agents, durable queue, hot index over object storage, quotas, signature sampling
• Failure mode: incident storms blind search; cost grows without owners
• Who pays for imperfection: on-call engineers (blind incidents), finance (unbounded spend), the storming team (sampled fidelity)

Security and audit → complete, intact, retained for years

• Constraint: no sampling, gap detection, tamper evidence, legal retention
• Strategy: separate lane with reserved ingest capacity, write-once storage, sequence numbers per source, independent verification
• Failure mode: silent gaps; deletion by an attacker with access
• Who pays: security (blind investigations), the company (compliance findings)

Analytics from logs → probably not a logging problem

• Constraint: long-range aggregations, stable schemas
• Strategy: emit metrics or structured events; if logs are the only source, extract metrics at ingest
• Failure mode: dashboards that scan terabytes per refresh
• Who pays: the platform (query load), everyone else (slow searches during dashboard refreshes)

2.2 When NOT to Build a Log Aggregation Platform#

• You're small. Under a few hundred GB/day, a managed logging service or a single search cluster is cheaper than a team. The tiering and quota machinery earns its keep at tens of TB/day.
• You need counts, rates and alerts. Use the metrics platform; a metric is bytes per minute, a log line is bytes per event.
• You need request-level causality across services. That's distributed tracing; link logs to traces by trace ID rather than reconstructing flows from text.
• You need business events with a schema. Use an event pipeline into a warehouse; logs are the wrong contract for data other teams depend on.

🎯 Staff Insight: "The cheapest log line is the one never written. Before I build a bigger platform, I'd find the ten streams that make up half the volume and ask whether anyone reads them."

2.3 What the Interviewer Leaves Underspecified#

Interviewers deliberately omit:

• Debugging vs audit — the completeness bar differs completely
• Who pays — shared cost hides the producers driving it
• Acceptable loss — "never lose a log" is unaffordable during a storm; what's the order?
• Retention — one number for everything vs per-stream tiers
• Structured vs free text — determines parsing cost and schema risk
• PII policy — whether logs may contain personal data at all, and how deletion requests apply

Staff engineers surface these and commit. Senior engineers assume them away and get surprised by the follow-up.

2.4 Precise Terminology#

🎯 Staff Insight: If the interviewer says "we can't lose any logs", ask: "Which logs? Audit logs, yes — reserved capacity, no sampling. Debug logs during a 10× storm from one service — I'd rather sample its repeated stack traces than blind every other team. Can we agree a loss order?"

3. Where the Design Splits#

Every logging decision has a technical side (index structures, buffer sizes, compression) and an organizational side (whose logs drop, who pays, who decides retention). Interviewers grade the second side.

3.1 Fault Line 1: Index Everything vs Query on Read#

The tension: A full-text index answers any search in sub-second time but costs ingest CPU and storage comparable to the raw data, replicated, on fast disks. Query-on-read stores compressed chunks cheaply and pays at query time by scanning.

Staff default: "Hot index for 7 days of in-quota data, with debug-level streams kept hot for only 24 hours. Everything written compressed to object storage, partitioned by tenant, stream and hour, with per-chunk metadata and token bloom filters. Queries route by time range; broad queries over cold data require a label selector so they can't scan the whole tenant."

When to deviate:

• Security investigations that need arbitrary search over months: index the audit lane for longer — it's a fraction of the volume.
• Small scale: a single indexed cluster with 14–30 days is simpler than two tiers.

🧭 Principal Move: "I'd publish the price of each tier per GB-month and the query latency teams should expect from it. Then retention becomes a choice teams make with their own budget, not a platform default everyone pays for."

❌ Common L5 Trap: "Elasticsearch with 90-day retention and daily indices." At 170 TB/day, that's on the order of tens of petabytes of replicated, indexed storage, most of it never queried after day two. The design isn't wrong — it's unaffordable, and the candidate didn't price it.

3.2 Fault Line 2: Backpressure — Block, Buffer or Drop#

The tension: When ingest exceeds capacity, logs must wait somewhere or be discarded. Waiting in the application blocks production; waiting on the host fills disks; waiting in the queue costs money; dropping loses evidence. The question is whose logs, where, in what order.

Staff default: "The application writes to local stdout files and never blocks on the network. The agent has bounded memory and disk buffers, honors 429s, and drops oldest when full — counting every drop per tenant. The queue retains 48 hours. Over-quota tenants are sampled by message signature in the hot path, while raw lines still go to object storage when its write path has capacity. Audit streams have reserved capacity at every stage and are never sampled."

When to deviate:

• Audit and security logs: block-or-fail is sometimes legally required — use a separate, synchronous, low-volume path for those specific events, not for all logs.
• Edge and mobile: client buffers are tiny and lossy; accept it and sample at the source.

🧭 Principal Move: "The loss order is an org policy: audit never, error-level last, debug first, and the storming tenant before anyone else. Security and engineering leadership sign it; the platform enforces it. Then a lost debug line during a team's own storm is policy, not an incident."

❌ Common L5 Trap: "Agents retry until the backend accepts, so nothing is lost." Unbounded retry means unbounded buffering — the agent fills the host's disk or memory and takes the application down with it. See Backpressure & Load Shedding.

3.3 Fault Line 3: Schema on Write vs Schema on Read#

The tension: Parsing at ingest gives typed fields, fast filters and compact columns — and a pipeline that breaks when formats change or two teams disagree on a field's type. Storing raw and parsing at query time is flexible and robust — and slow, with every query re-parsing.

Staff default: "Hybrid with per-stream namespaces. Teams register the fields they want typed; those are parsed at ingest and indexed in the hot tier. Everything else stays in the raw body, searchable as text. A field that fails to parse is kept raw with a parse-error tag — never dropped. No shared dynamic mapping, so one team's new field can't push a shared index past its field limit."

When to deviate:

• Highly regular logs (load balancer access logs): full schema on write, columnar, typed — they're effectively events.
• Third-party software logs with unstable formats: raw only, with query-time parsing helpers.

🎯 Staff Insight: "Elasticsearch's default limit is 1,000 mapped fields per index for a reason. In a shared index, the 1,001st field from any team breaks ingest for everyone. Per-stream schemas turn a shared failure into a local one."

❌ Common L5 Trap: "We parse JSON and index all fields automatically." It works until one service logs a dynamic map — user IDs as keys — and mints thousands of fields an hour, or two services disagree on whether status is a string. Logs get rejected for tenants who did nothing wrong.

3.4 Fault Line 4: Retention Tiers and Who Pays#

The tension: Retention is the largest cost multiplier: every extra day multiplies every byte. Teams want long retention by default because it's free to them; the platform wants short retention because it pays. Neither has the right incentive.

Staff default: "Three tiers. Hot, indexed: 7 days for info and above, 24 hours for debug. Warm, object storage: 30 days default, up to 90 per stream if the team pays. Archive: audit streams only, write-once, retention set by security and legal. Every stream's bytes ingested and GB-months retained are reported per team monthly, with the top streams by cost and their query counts — 'you paid $14K for this stream; it was queried twice.'"

When to deviate:

• Regulated environments: retention minimums set by policy override team choice.
• Early-stage platforms: one tier, short retention; tiering arrives with scale.

🧭 Principal Move: "Chargeback changes behavior more than any compression algorithm. The first month teams see their logging bill next to their query counts, volume drops — I'd plan for that and measure it."

❌ Common L5 Trap: "Keep 30 days for everything; storage is cheap." Object storage is cheap; indexed, replicated SSD storage for 30 days of 170 TB/day is not. And "cheap" multiplied by a volume that doubles yearly is a line item finance will notice.

3.5 Fault Line 5: Where PII Redaction Happens#

The tension: Redacting at the source is most effective but depends on every team's discipline. Redacting in the pipeline is centralized but sees only what patterns can recognize. Redacting at query time leaves secrets in storage. And once data is in compressed, immutable chunks, deletion means rewriting files.

Staff default: "Defense in depth. Logging libraries mask known sensitive fields by type. Agents run fast pattern redaction for credentials and card numbers before anything leaves the host. Processors apply per-stream field allowlists. Daily canaries inject fake secrets through real services and alert if any reach storage unmasked. Storage is partitioned by tenant and hour so a purge rewrites a bounded set of chunks, and the hot index supports delete-by-query for the 7-day window."

When to deviate:

• Debugging that requires sensitive values (e.g., payment flows): log a reference or a keyed hash, not the value; give a separate, access-controlled store for the rare cases that truly need raw data.
• Regions with strict data protection: keep logs in-region and apply stricter allowlists.

🧭 Principal Move: "Logs are a database the whole company writes to without review. I'd put logging under the data classification standard, give security a per-service report of emitted fields, and track redaction canary pass rate as a compliance metric."

❌ Common L5 Trap: "We mask PII in the search UI." The secret is still in the index, the replicas, the object store, and any export. A breach or an over-privileged query path exposes it, and a deletion request means finding it everywhere.

4. When It Breaks#

4.1 The Debug Flag — One Team Triples Platform Volume#

t=0:       A team enables DEBUG in production to chase a bug. Their volume: 60 MB/s → 2.4 GB/s.
t=+1min:   Total ingest 2 GB/s → 4.3 GB/s. No per-tenant quotas enforced in processing.
t=+4min:   Hot-tier indexers at 100% CPU. ingest.lag_p99_s for all tenants: 8s → 6 min.
t=+10min:  Queue depth rising 2 TB/min. Live tail lags. Two unrelated incidents begin.
t=+25min:  On-call finds the stream by bytes; asks the team to revert. Lag drains over 40 min.

Detection: tenant.ingest_bytes_s top-N vs quota; ingest.lag_p99_s{in_quota}; indexer.cpu_util.

Mitigation: enforce quota in processing with signature sampling for the over-quota tenant; route its excess directly to object storage, skipping the hot index.

Prevention: quotas enforced by default at gateway and processor; debug level kept hot for 24 hours only; a "debug window" feature that lets a team raise its quota for 30 minutes on a named set of pods, with cost shown up front.

Owner: logging platform on-call; the producing team owns its volume.

4.2 The Full Disk — An Agent Takes Down Its Host#

t=0:       Ingest gateway in one zone rejects connections (cert expired).
t=+5min:   Agents in that zone buffer to disk. Buffer limit misconfigured as "unlimited" on
           an older agent version deployed to 4,000 nodes.
t=+3h:     Node disks at 95%. Kubelet starts evicting pods for disk pressure.
t=+3h10m:  Production services in the zone lose capacity. A logging outage became a
           production outage.

Detection: agent.buffer_bytes{node}, agent.buffer_pct_of_limit, node disk pressure events.

Mitigation: cap agent disk buffers at a fixed size (e.g., 1 GB) and a fraction of free disk; on limit, drop oldest and count; rotate the certificate.

Prevention: agent config enforced by a fleet policy, not per-deployment; certificate expiry alerts 30 days ahead; a chaos test that blackholes the gateway for one zone monthly.

Owner: logging platform (agent fleet and gateway).

4.3 The Mapping Explosion — One Service Breaks the Shared Index#

t=0:       A service starts logging a map keyed by customer ID as a JSON object.
t=+20min:  Shared hot index gains 9,000 new fields; hits its field limit.
t=+21min:  Every write with a new field to that index is rejected; with dynamic mapping,
           12 other services sharing the index start losing logs.
t=+1h:     Teams notice "missing logs"; platform finds the culprit by field growth rate.

Detection: index.field_count growth rate; ingest.rejected_total{reason=mapping}; per-stream new-field rate.

Mitigation: move the stream to its own index or fold unknown fields into the raw body; blocklist the offending field path.

Prevention: no shared dynamic mapping; per-stream registered schemas; unknown fields stored raw; a per-stream limit on parsed fields.

Owner: logging platform (schema policy); producing team (log format).

4.4 Secrets in Logs — Six Weeks of Bearer Tokens#

t=0:       A new HTTP client library logs full request headers at INFO on retry.
t=+6 wks:  Security finds Authorization headers in a log search result.
t=+6 wks:  Tokens for ~2M sessions sit in the hot index, its replicas, 42 days of object
           storage chunks and two analytics exports.

Detection: redaction canaries (a fake token sent through real services daily; alert if found unmasked); periodic scans for credential patterns in stored data.

Mitigation: revoke affected tokens; delete-by-query in the hot tier; rewrite affected chunks in object storage — bounded because chunks are partitioned by tenant and hour; purge exports.

Prevention: agent-level pattern redaction for authorization headers; logging library denies header logging by default; canaries per language runtime.

Owner: security (rules, response), logging platform (purge tooling), producing team (library usage).

4.5 Silent Loss — A Parse Error Path That Drops Lines#

t=0:       A processor deploy changes JSON parsing; lines with invalid UTF-8 now throw.
t=+0:      The exception handler drops the line and increments a counter nobody alerts on.
t=+5 days: An incident: the on-call can't find errors from one service. 3% of its lines
           (those with binary payload fragments) have been dropped for five days.

Detection: end-to-end counts: lines read by agents vs lines stored, per stream, reconciled hourly; processor.dropped_total{reason} with alerts.

Mitigation: fix parser to keep unparseable lines raw with a tag; replay the queue for the last 48 hours (the rest is lost).

Prevention: invariant: the processor never drops a line without a counted, alerted reason; canary streams with malformed content in every deploy's test suite.

Owner: logging platform.

4.6 The Query of Doom — A 90-Day Unfiltered Search#

An engineer runs a regex across all services for 90 days during an incident. The query fans out to hundreds of thousands of chunks and saturates the query tier; every other engineer's searches queue behind it during the incident. Mitigation: per-tenant and per-user limits on bytes scanned and concurrent queries; require a label selector for queries beyond the hot window; queue long queries as async jobs with results delivered later.

Detection: query.bytes_scanned{user}, query.queue_wait_p99.

Owner: logging platform (query limits).

4.7 Operational Reality Matrix#

🎯 Staff Insight: The platform's health metric is in-quota ingest lag, not overall lag. "Overall lag will spike whenever a team storms, and it should — they're over quota. I page on lag for tenants within quota, because that number only moves when the platform is the problem."

5. Scorecard#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Deep Elasticsearch tuning ≠ log platform design. Shard sizing doesn't address loss policy, quotas or cost attribution.
• "We use Kafka" ≠ backpressure. A queue moves the buffer; it doesn't decide what happens when the buffer fills.
• Compression ratios ≠ cost control. Chargeback and volume reduction at source usually save more.
• Regex redaction ≠ PII safety. Without canaries and a deletion path, it's a hope, not a control.

6. The 45 Minutes, Phase by Phase#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Index internals — "inverted index for the hot tier; columnar chunks for warm."
• UI design — say what queries look like, not the screens.
• Agent implementation details — name the bounds, skip the file-tailing mechanics.
• Log formats per language — "structured JSON encouraged; raw accepted."
• Exactly-once ingestion — at-least-once with dedupe by (agent, file, offset) where it matters; say so in one line.

6.4 Follow-Up Questions to Expect#

1. "One team's volume goes 10× during an incident. What happens to everyone else's logs?"
2. "The ingest tier is down for an hour. Walk me through every buffer between the app and storage."
3. "What do you index, for how long, and what does 90 days cost?"
4. "Two services log status with different types. What breaks?"
5. "A secret was logged for six weeks. How do you delete it everywhere?"
6. "How do you know logs aren't silently missing?"
7. "Security needs a year of retention for auth logs. How does that differ from debug logs?"

7. Practice Rounds#

Drill 1: The Opening#

Prompt: "Design a centralized logging system for our company."

Drill 2: The Ingest Outage#

Prompt: "The ingest tier is unreachable for one hour. Walk me through every buffer between the app and storage."

Drill 3: Make It Concrete — Size and Price the Tiers#

Prompt: "Size the storage tiers for 2 GB/s, 7 days hot, 90 days warm. Roughly what does it cost?"

Drill 4: Index or Scan?#

Prompt: "Why not just index everything? Engineers want fast search over all 90 days."

Drill 5: The Noisy Tenant#

Prompt: "A team turns on debug logging in production and their volume goes from 60 MB/s to 2.4 GB/s. Everyone else's logs are delayed. Fix it."

Drill 6: Cardinality and Mapping Explosions#

Prompt: "A service starts putting user IDs into a label — or into JSON keys. What happens, and how do you prevent it?"

Drill 7: Build vs Buy#

Prompt: "Should we run our own logging platform or use a vendor?"

Drill 8: Changing a Parsing Rule Without Losing Logs#

Prompt: "You need to change the parser for 400 services' access logs to a new format. How do you ship it?"

Drill 9: Cut the Bill by 40%#

Prompt: "Finance says logging costs too much. Cut it by 40% without hurting incident response."

Drill 10: Multi-Region#

Prompt: "We run in four regions. How does logging work across them?"

8. Incident Walkthroughs#

Deep Dive 1: Peak-Traffic Incident — Logs Go Dark During the Outage#

Context: A database failover causes errors across 600 services. Log volume jumps from 2 GB/s to 11 GB/s within two minutes. Ingest lag for all tenants climbs to 14 minutes, and incident responders can't see current errors. The on-call escalates to you.

Questions to Surface First:

• Where is the bottleneck — gateway, queue, processors or hot-tier indexing?
• Is the surge broad (600 services) or dominated by a few?
• Are quotas being enforced, or are all tenants bursting at once?
• Are responders' queries themselves loading the hot tier?

Typical L5 Approach: Scales the indexing cluster. Shard rebalancing adds load; lag gets worse for 20 minutes before it improves.

Staff Approach: Finds the processors keeping up but hot-tier indexing saturated. Since every tenant is bursting, per-tenant quotas don't help — so applies the platform-wide loss order: debug and info streams skip the hot index (still written to object storage), error and warn keep flowing. Hot-tier lag drops to 20 seconds for error-level logs within 3 minutes.

Principal Approach: Recognizes that broad incidents create correlated bursts that per-tenant quotas can't handle and that log volume during failures is predictable. Builds a platform-level "incident mode" — pre-agreed level-based shedding — and drives a program to collapse repeated error logs at the source (rate-limited loggers per message template).

Key Takeaway: "When everyone storms at once, quotas don't help — a pre-agreed level-based loss order does."

What clears the Staff bar:

• Localizes the bottleneck before scaling
• Uses a loss order that preserves error-level logs
• Pushes the fix to the source: rate-limited repeated errors

Deep Dive 2: Silent Failure — Three Days of Missing Logs From One Region#

Context: During a post-mortem, an engineer notices that logs from one region's newest node pool are missing for the last three days. No alerts fired. Ingest lag, error rates and queue depth all look normal.

Questions to Surface First:

• Are the agents on those nodes running, and what version?
• What do agent-side counters say about lines read vs shipped?
• Is there any end-to-end reconciliation between agents and storage?
• Did anything change for that node pool three days ago?

Typical L5 Approach: Restarts agents on the node pool; logs start flowing. The three days are lost and the cause is unknown.

Staff Approach: Finds the node pool uses a new container runtime that writes logs to a different path; the agent's file discovery doesn't match it, so the agent is healthy and shipping nothing. Fixes discovery; recovers what's still on the nodes' rotated files. Adds end-to-end reconciliation: per node, expected log activity versus received; alerts on nodes with running pods and zero received lines.

Principal Approach: Treats "absence of data" as a first-class signal across observability. Every node and every service has an expected heartbeat in logs, metrics and traces; silence pages. Node pool changes require an observability checklist.

Key Takeaway: "A healthy agent shipping nothing looks exactly like a quiet service. Alert on silence, not just on errors."

What clears the Staff bar:

• Recognizes missing data needs its own detection
• Adds end-to-end reconciliation and canaries
• Gets the change process for node pools to include observability

Deep Dive 3: Large-Customer Onboarding — A Team With 400 TB/Day#

Context: The CDN team wants to move edge request logs onto the platform: 400 TB/day raw, more than double the platform's current total. They want 30 days of search and dashboards over status codes and latencies.

Questions to Surface First:

• Are these logs searched for needles, or aggregated for dashboards?
• What fraction is ever queried? Do they need every 200 response?
• Is the format regular enough to treat as structured events?
• Who pays, and what's their budget?

Typical L5 Approach: Triples the hot cluster and onboards them into the shared index. Ingest saturates during their daily peak; every other tenant's lag rises.

Staff Approach: Recognizes these are events, not debug logs: fixed schema, aggregation-heavy queries. Routes them to a dedicated columnar pipeline — typed columns, high compression, fast aggregations — with metrics extracted at ingest for dashboards. Errors (4xx/5xx) and a 1% sample of 2xx are kept searchable for 30 days; full data in object storage for 7 days for investigations. Their quota and cost are separate.

Principal Approach: Uses the onboarding to define a platform tier for high-volume structured events, with its own pricing, and sets the rule that dashboards over logs must be served by extracted metrics.

Key Takeaway: "High-volume regular logs are events. Give them a columnar lane and extracted metrics, not the shared search index."

What clears the Staff bar:

• Classifies the data by query pattern before choosing storage
• Isolates the new tenant's capacity and cost
• Uses sampling and metric extraction instead of indexing everything

Deep Dive 4: Post-Mortem — Card Numbers in Logs for 41 Days#

Context: A compliance scan finds full payment card numbers in log search results. A checkout service started logging raw request bodies on validation errors 41 days ago. The data is in the hot tier, object storage, and a weekly export to the analytics warehouse. You're leading the post-mortem.

Questions to Surface First:

• Why didn't pipeline redaction catch card numbers?
• Where exactly does the data exist — tiers, replicas, exports, backups?
• Who accessed those log streams during the window?
• Can we purge object storage without rewriting the entire 41 days?

Typical L5 Approach: Adds a regex for 16-digit numbers to the pipeline and deletes the hot index for those days.

Staff Approach: Finds the card numbers were embedded in a URL-encoded body the regex didn't decode. Adds Luhn-validated detection on decoded content, redaction in the logging library for request bodies, and a canary that sends a test card number through checkout daily. Purges hot-tier documents by query, rewrites only the affected tenant-and-hour chunks in object storage, deletes the export partitions, and produces an access report for compliance.

Principal Approach: Treats this as a gap in data governance: logs had no classification and no owner for what fields services emit. Puts logging under the data classification standard, requires that payment services' logs pass through stricter allowlist-only redaction, and makes canary pass rate a compliance control.

Key Takeaway: "Redaction you don't test with canaries is redaction you hope works. And design storage so you can delete from it."

What clears the Staff bar:

• Finds why redaction failed, not just adds another rule
• Purges every copy, including exports, with bounded rewrites
• Adds continuous verification with canaries

Deep Dive 5: Multi-Region Expansion — Residency for EU Logs#

Context: The company is launching EU data residency. Logs from EU workloads may contain personal data and must stay in the EU. Today all logs flow to a central US platform.

Questions to Surface First:

• Which services process EU personal data, and do their logs contain it?
• Can we run a full regional stack, or only regional storage?
• How do engineers outside the EU search EU logs for incidents?
• Where do derived artifacts go — extracted metrics, alerts, exports?

Typical L5 Approach: Deploys a second cluster in the EU and routes EU logs there. Leaves the analytics export and alert notifications, which include log excerpts, flowing to the US.

Staff Approach: Full regional stack in the EU: agents, gateway, queue, processors, hot tier and object storage. Federated queries from the global frontend execute in the EU and return results to authorized users under an access policy; results aren't cached outside the EU. Extracted metrics without personal data can flow globally; alert notifications carry links, not excerpts.

Principal Approach: Defines residency as a tenant attribute honored by every observability system — logs, traces, metrics labels, alert payloads — with automated marker tests proving compliance, and a single access-policy layer for cross-region queries.

Key Takeaway: "Residency includes everything derived from logs — alerts, exports and caches, not just the index."

What clears the Staff bar:

• Inventories derived data, not just primary storage
• Uses federated queries instead of moving data
• Proves residency with tests

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Explain why log volume spikes during incidents and why the platform must stay useful then
• Design bounded buffering at every stage with a written loss order that never blocks applications
• Choose what to index (hot window) and what to query on read (object storage), and price both
• Enforce per-tenant ingest quotas, query limits and cardinality limits, with signature sampling under overload
• Contain schema conflicts with per-stream schemas and raw fallback
• Layer PII redaction, verify it with canaries, and design storage so purges are bounded
• Size and price storage tiers, and attribute cost to producing teams
• Detect silent loss with end-to-end reconciliation and silence alerts

The Bar for This Question#

Mid-level (L4): Ships logs from hosts to a search cluster with dashboards. Works at small scale. No backpressure story, no quotas, one retention, no redaction.

Senior (L5): Adds agents, Kafka, Elasticsearch with daily indices, retention policies and some parsing. The gap: "Kafka buffers it" ends the backpressure discussion; indexes everything for the full retention without pricing it; no per-tenant isolation; shared dynamic mapping; PII left to developer discipline. The design works on a quiet day and fails during the first incident storm.

Staff+ (L6): Frames the problem as staying useful during the worst hour at a bounded cost within the first five minutes. Bounds every buffer and writes down the loss order. Indexes the hot window and queries the rest on read, with prices. Isolates tenants with quotas, query limits and cardinality limits. Contains schema failures per stream. Layers redaction with canaries and a deletion path. Names who pays — storming tenants lose fidelity on their own repeated logs; everyone else pays nothing. The interviewer should learn something from the answer.

10. Hot Takes#

10.1 Most Logs Are Never Read#

The Staff position: Index what's searched; keep the rest cheaply or not at all. Volume reduction at the source beats storage engineering.

Why this matters in interviews: Pricing retention and admitting most bytes are cold is a Staff signal.

10.2 "Never Lose a Log" Is a Promise You Can't Afford#

The Staff position: Promise completeness for audit; publish a loss order for everything else.

Why this matters in interviews: It shows you design the failure behavior instead of pretending there isn't one.

10.3 Logs Are a Bad Metrics System#

The Staff position: Extract metrics at ingest if you must; better, emit them at the source.

Why this matters in interviews: Redirecting analytics to metrics is a strong scoping move.

10.4 Shared Dynamic Mapping Is a Multi-Tenant Bug#

The Staff position: Any shared index where one tenant's new field can reject another tenant's logs is an isolation failure. Per-stream schemas and raw fallback aren't polish; they're tenancy.

Why this matters in interviews: It's the failure Senior designs don't see until it happens.

10.5 Chargeback Saves More Than Compression#

The Staff position: Teams who see their logging bill next to their query counts cut volume. No codec delivers that.

Why this matters in interviews: It shows you treat cost as an organizational problem with a technical assist.

11. Beyond Staff: The Principal View#

Why L7 Sees This Problem Differently#

The Staff engineer builds a log platform that stays useful during incidents, isolates tenants and prices its tiers. The Principal engineer notices that logs, metrics and traces are three pipelines with three agents on every node, three bills, three retention policies and three places personal data can leak — and that observability spend is growing faster than revenue. The L7 problem is observability as one budget with one collection layer: a shared agent and pipeline, per-signal storage, consistent tenancy and redaction, and a cost target the organization manages to.

The Org-Level Fault Line#

One observability pipeline vs per-signal stacks.

🧭 Principal Move: "One collection layer — agent, gateway, redaction, tenancy, quotas and cost attribution — shared by logs, metrics and traces. Backends stay specialized. Every team gets one observability bill with three lines."

Cost Model#

Assumptions: object storage at ~$0.02/GB-month order of magnitude; ~10× compression; hot tier on replicated fast storage; fully loaded engineer ~$250K/year. Illustrative ranges.

The pricing insight: the hot index usually costs more than the warm tier despite holding a fraction of the days. Shrinking the hot window for debug-level streams and reducing volume at the source typically saves more than any storage-engine migration.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-OBS-003: Logging Standard
Status: Approved   Owners: Observability Platform + Security

Scope
  Every service and job that emits logs in any environment.

MUST
  1. Emit logs to stdout or the platform agent; never ship directly from
     application code to storage.
  2. Never block request handling on logging.
  3. Use only the standard label set; high-cardinality values go in the body.
  4. Pass through platform redaction; payment and identity domains use
     allowlist-only fields.
  5. Route security and audit events to the audit lane.

SHOULD
  1. Emit structured JSON with a registered schema per stream.
  2. Rate-limit repeated messages per template in the logging library.
  3. Emit metrics for anything counted on a dashboard.

Exceptions
  Filed with the Observability Platform; security review for redaction
  exceptions; time-boxed to two quarters.

Success metrics
  - In-quota ingest-to-searchable p99: under 30 s
  - Redaction canary pass rate: 100%
  - Logging cost per 1M requests served: tracked per team, trending down
  - Unplanned loss for in-quota tenants: under 0.1%

What I'd Tell the VP#

"Logging is our second-largest infrastructure cost after compute, and it's growing faster than traffic because nobody who writes logs sees the bill. During our last two major outages, the logging platform fell behind exactly when we needed it. I'm proposing three things: per-team quotas and monthly cost reports so teams own their volume, a tiered design where only the last week is fast and expensive, and a pre-agreed policy for what gets dropped during a company-wide incident. I expect a 30–40% cost reduction within two quarters, mostly from teams cutting volume once they see it, and logs that stay searchable during outages."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll build a great logging platform" — correct, but ignores the metrics and tracing agents on the same nodes.
• "Teams can see their usage in a dashboard" — good, but without budgets or targets nothing changes.
• "We'll add an EU cluster" — no mention of alert payloads, exports and caches derived from logs.

Appendices

Appendix A: Mechanics in Depth#

A.1 Agent Buffering Loop#

config:
  mem_buffer_max = 64MB
  disk_buffer_max = min(1GB, 10% of free disk)
  batch_max = 1MB or 1s

loop:
  read new lines from discovered files (respect rotation, track offsets)
  redact_patterns(line)               # tokens, keys, Luhn-valid card numbers
  append to mem_buffer
  if mem_buffer full: spill oldest batch to disk_buffer
  if disk_buffer full: drop oldest batch; dropped_lines[tenant] += n
  send batches oldest-first:
     204 -> commit offsets
     429 -> backoff(Retry-After, jitter); keep buffered
     5xx/timeout -> exponential backoff with jitter; keep buffered

A.2 Signature Sampling#

signature(line) = hash(template(line))    # numbers, IDs, hex stripped
per tenant, per minute:
  if tenant under quota: keep all
  else:
    count[sig] += 1
    if count[sig] <= 100: keep full line (hot + warm)
    else: warm only (if capacity), emit summary record every 10s:
          { sig, example, suppressed_count }
audit streams: never sampled

A.3 Query Routing#

route(query):
  require label selector with at least one of {service, tenant stream}
  split time range into hot (<= 7d) and warm (> 7d) parts
  hot part  -> index search, limit by tenant concurrency
  warm part -> list chunks by (tenant, stream, hour) -> prune by bloom filter
               -> parallel scan, cap bytes_scanned per query
               -> if over cap: return partial + offer async job
  merge, sort by ts, apply limit

Appendix B: Data Model#

CREATE TABLE tenants (
  tenant_id           TEXT PRIMARY KEY,
  cost_center         TEXT NOT NULL,
  ingest_quota_bps    BIGINT NOT NULL,
  burst_bytes         BIGINT NOT NULL,
  max_labels          INT NOT NULL DEFAULT 12,
  max_query_bytes     BIGINT NOT NULL
);

CREATE TABLE stream_policies (
  tenant_id     TEXT NOT NULL REFERENCES tenants,
  selector      TEXT NOT NULL,          -- e.g. {service="checkout",level="debug"}
  hot_days      INT NOT NULL DEFAULT 7,
  warm_days     INT NOT NULL DEFAULT 30,
  schema_ref    TEXT,                   -- registered parser/schema version
  redaction     TEXT NOT NULL DEFAULT 'standard',  -- standard | allowlist
  PRIMARY KEY (tenant_id, selector)
);

-- chunk metadata (object storage index):
-- chunks(tenant_id, stream_hash, hour, chunk_id, t_min, t_max, bytes,
--        bloom_ref, schema_ref)  -- object key: tenant/stream_hash/yyyy/mm/dd/hh/chunk_id

Appendix C: Coordination Mechanisms#

C.1 Ingest Under Quota#

C.2 Quick Comparison#

Appendix D: API Contract & Client Behavior#

• Agents batch up to 1 MB or 1 second, gzip, and retry on 429/5xx with jittered backoff; they never drop on 429.
• Agents report dropped_lines_total, buffer_bytes and lines_shipped per tenant as metrics.
• Applications log to stdout; structured JSON preferred, one record per line, timestamps in UTC with milliseconds.
• Labels: a fixed, small set per stream; request and user IDs go in the body.
• Queries beyond the hot window require a label selector; large scans run as async jobs.

Appendix E: Observability#

Core metrics:

• ingest.lag_p99_s{in_quota}, ingest.lag_p99_s{level}
• agent.dropped_lines_total{tenant}, agent.buffer_pct_of_limit, agent.lines_shipped{node}
• tenant.ingest_bytes_s, tenant.quota_exceeded, sampling.suppressed_lines_total
• index.field_count, ingest.rejected_total{reason}, processor.dropped_total{reason}
• query.bytes_scanned{user}, query.queue_wait_p99
• redaction.canary_pass, cost.per_gb_by_team

Critical alerts:

Debugging the silent failure: ingest lag and error rates look fine when data never arrives. Reconcile lines read by agents against lines stored, alert on silent streams and nodes, and run a canary log line through every node pool.

Appendix F: Scale Evolution#

What you don't build on day one: tiered storage, chargeback, signature sampling, federated queries, an audit lane with tamper evidence. Each has a trigger in Section 11.

Appendix G: Multi-Tenancy, Fairness & Cost#

• Quotas: bytes per second with burst per tenant; raised through a budget request, not a ticket to the platform team.
• Query fairness: concurrent queries and bytes scanned per tenant and per user; long scans queued as async jobs.
• Cardinality: labels per stream and distinct values per label per day capped; violations rejected with a clear error.
• Cost attribution: bytes ingested, GB-months per tier and bytes scanned reported per team monthly, with top streams and their query counts. Rehearse the sizing with the back-of-envelope calculator and the cost estimator.