Technologies referenced in this case study: Apache Kafka · ZooKeeper & etcd · Redis · PostgreSQL · DynamoDB

Related: Idempotency · Stream Processing · Task Scheduling · Notification Systems · Scaling Writes · Degraded Mode · Long-Running Processes · Data Pipelines · Consistency Models · Sharding

How to Use This Case Study#

This case study is organized for the interview first and for reference second. Read it front-to-back once; afterwards, return to the fault lines and deep dives that expose your weak spots. Broker internals (segments, ISR, the controller) live in the Kafka guide — this page is about the design problem: what the queue promises, to whom, and who owns the messages nobody can process.

Without a designed delivery contract:
t=0:        Checkout publishes "order_placed" to the queue. Broker acks after
            writing to the leader's page cache only.
t=+40ms:    Leader host loses power. Follower had not fetched the message yet.
t=+12s:     Follower promoted. The message never existed. Producer already returned 200.
t=+3 days:  Customer emails support: "no confirmation". Nobody can find the message.
            Meanwhile, a consumer that takes 45s per message has a 30s visibility
            timeout — every slow message is processed twice. 3% of customers get
            duplicate emails. The team "fixes" it by raising the timeout to 12h,
            so the next consumer crash strands 8,000 messages for 12 hours.

With an explicit contract:
t=0:        Producer sends with idempotence key. Broker acks only after the message
            is on 2 of 3 replicas in 2 AZs (p99 ~8ms).
t=+40ms:    Leader dies. Follower already has the message. Promoted in ~5s.
t=+5s:      Producer retries in-flight batch. Broker dedups by (producer_id, seq).
t=+6s:      Consumer leases message for 60s, heartbeats every 20s while working.
t=+51s:     Consumer finishes, acks. No duplicate. Lease never expired.
t=+1 day:   One malformed message failed 5 times → DLQ. Owning team paged
            by dlq.oldest_message_age > 1h. Fixed and redriven the same day.

Executive Summary

If you only read one section, read this. Everything in this case study flows from the contrast below.

What This Interview Actually Tests#

A message queue is not a storage question. Everyone can append bytes to a file.

It is a contract question: what exactly has the producer been promised when it gets an ack, what exactly has the consumer been promised when it gets a message, and who is accountable for the messages that fall between those two promises. It tests:

• Whether you define durability at ack time in terms of replicas and failure domains, not adjectives
• Whether you know that ordering and parallelism are the same knob turned in opposite directions
• Whether you treat redelivery as normal operation and push idempotency to the consumer, by contract
• Whether you design for the backlog — the moment consumers fall behind is when the queue earns its keep or loses data
• Whether the DLQ has an owner, an alert and a redrive path, or is a place messages go to die

The key insight: A queue does not remove failure; it moves failure in time. Every message you accept is a promise you'll keep later, under conditions you can't see now. Staff candidates design the conditions under which that promise is kept — ack quorum, lease duration, retry budget, retention vs maximum tolerable lag — and name who gets paged when it isn't.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

Three intents produce three different systems. Name them, then commit.

🎯 Staff Move: "I'll design the work-queue intent — durable async jobs between services, at-least-once, per-message ack, retries and DLQ — because that's where delivery semantics, leases and poison handling all bite. I'll build it on a partitioned replicated log so we keep replay and can serve the event-log intent later with the same storage. If you meant fan-out for live updates, that's a different, much lighter system and I'd drop durability entirely."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Naming how real systems settled these fault lines shows you've studied operations, not just APIs.

Amazon SQS — The Lease Model as a Public Contract#

SQS standard queues are documented as at-least-once with best-effort ordering; a received message becomes invisible for a visibility timeout (default 30 seconds, configurable up to 12 hours) and reappears if not deleted. A redrive policy moves a message to a dead-letter queue after maxReceiveCount receives. FIFO queues add ordering within a message group ID and deduplicate by a deduplication ID within a 5-minute window. Retention defaults to 4 days, maximum 14.

Staff insight: SQS makes every fault line in this case study a visible parameter — lease length, receive count, group key, dedup window, retention. In an interview, naming those knobs and their defaults signals you've run one; explaining why the dedup window is only 5 minutes (dedup state is expensive, and producer retries happen within seconds) signals Staff.

Uber — A Consumer Proxy in Front of Kafka#

Uber has written publicly about a Kafka consumer proxy that pulls from Kafka and pushes to service endpoints over gRPC. The proxy tracks per-message acknowledgement, retries failed messages through retry queues, routes exhausted ones to a DLQ, and lets a service process a single partition with many parallel workers — decoupling consumer parallelism from partition count.

Staff insight: This is the "log + lease layer" model. The log gives durability and replay; the proxy gives work-queue semantics and isolates poison messages without stalling the partition. It's the shape this case study's design converges on, and it's the answer to "but Kafka can't do per-message ack."

Slack — Durable Ingestion in Front of a Fast Job Queue#

Slack has described re-architecting its asynchronous job queue after a Redis-based queue hit memory exhaustion during a backlog and lost the ability to accept jobs. The redesign put Kafka in front as a durable buffer, with a relay feeding jobs into Redis for execution, so a backlog grows on disk rather than in RAM.

Staff insight: The failure was not "Redis is bad" — it was the backlog had nowhere to go. Queues fail at exactly the moment they're needed most: when consumers are slow. A Staff design states where the backlog lives at 10× normal depth and how long it can grow before something breaks.

What Interviewers Probe#

System Architecture Overview#

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The walkthrough below is a 45-minute script. The words in italics are meant to be said out loud.

Phase 1: Requirements & Framing (2–3 minutes)#

Open by splitting the intents, then commit.

"Before I draw anything — 'message queue' covers three different systems. A work queue where each job is done once and acked individually; an event log that many teams replay; and ephemeral fan-out where old messages are worthless. I'll design the work queue as an internal platform: durable, at-least-once, per-message ack, retries, DLQ, and opt-in per-key ordering. I'll put it on a partitioned log so replay is available later."

Then pin down the numbers that change the design:

"I'll treat end-to-end latency — produce to consume — as p99 under 100ms when consumers are healthy, and I won't optimize below that; durability and isolation matter more here."

Phase 2: Core Entities & API (1–2 minutes)#

POST /queues/{q}/messages                       # produce (batch up to 500 / 1 MB)
  body: [{ message_id?, key?, payload, headers, delay_seconds? }]
  → 200 { results: [{ message_id, status: "accepted" | "duplicate" }] }

POST /queues/{q}/receive                        # long-poll
  body: { max_messages: 10, wait_seconds: 20, lease_seconds: 60 }
  → 200 [{ message_id, receipt, payload, attempt, enqueued_at }]

POST /queues/{q}/ack        { receipts: [...] }
POST /queues/{q}/nack       { receipt, retry_after_seconds? }
POST /queues/{q}/extend     { receipt, lease_seconds }   # heartbeat
POST /queues/{q}/redrive    { from: dlq, rate_per_sec, filter? }   # operator API

"Two details matter: the receipt is per-delivery, not per-message — a consumer whose lease expired can't ack a message someone else now holds. And produce returns 'duplicate' rather than an error when a producer retries a message ID we've already accepted, so the producer's retry loop is trivially safe."

Phase 3: High-Level Architecture (≤5 minutes)#

Staff candidates spend under five minutes here. Draw the tiers, state what each one owns, and move on.

"Front-end is stateless: auth, per-tenant quotas, and routing by hash(key) mod partitions. Storage is a replicated log — I'll lean on Kafka-style internals and not redesign segment files. The interesting tier is the dispatcher: it turns a log into a work queue by tracking per-message leases on top of the offset, so one slow message doesn't stall a partition and consumers can scale past partition count. Metadata — partition maps, leaders, queue config — lives in a small Raft group, like etcd."

Phase 4: Transition to Depth (1 minute)#

The sentence that steers the interviewer toward your strongest ground:

"The boxes are standard. The design lives in three places: what the producer ack means when a leader dies, what happens between a consumer receiving a message and acking it, and what happens to messages that can never succeed. I'd like to go through those in that order, then cover multi-tenancy. Is there one you'd rather start with?"

Phase 5: Deep Dives (25–30 minutes)#

Deep dive 1 — The ack path (8 min).

"The ack goes back when the message is on the leader and at least one follower in another AZ — min_in_sync = 2 of RF 3. That costs one cross-AZ round trip, ~1–2ms, plus batching; p99 lands around 5–15ms. If the in-sync set drops below 2, I reject produces rather than ack with one copy: producers see errors, which is honest, instead of success, which is a lie. Producer idempotence — producer ID plus sequence number per partition — means a retry after a lost ack is deduplicated by the leader."

"On fsync: I don't fsync per message. Two copies in two AZs' page caches is stronger than one copy fsynced, because correlated failure across AZs is much rarer than a single host losing power. For the critical tier, I'd add periodic fsync — every 100–500ms — to bound loss under a full-region power event."

Deep dive 2 — The lease (10 min).

"A receive grants a lease of 60s by default. Long-running consumers heartbeat via extend every lease/3 — 20s — so a 10-minute job never gets redelivered while its worker is alive, but a crashed worker's message comes back within 60s. The receipt handle includes a lease generation; an ack with a stale generation is rejected and counted in ack.stale_receipt_rate, which is my early warning that leases are too short."

"The dispatcher tracks leases per partition. Its state — which offsets are acked, which are leased until when — goes to a compacted ack log, so a dispatcher failover rebuilds in seconds. The committed offset for the partition is the lowest un-acked message; everything below it is done. If one message is stuck at attempt 3 while 50,000 after it are acked, the gap is tracked as a sparse set, not by blocking."

Deep dive 3 — Poison messages and DLQ (6 min).

"Each nack or lease expiry increments attempt. Retries are delayed, not immediate: 10s, 1m, 10m, 1h — implemented as delay tiers rather than per-message timers, so the delay mechanism is just more queues. After attempt 5, the message moves to the DLQ with its full history: error strings, consumer IDs, timestamps. The DLQ belongs to the consuming team, alerts on age, and has a rate-limited redrive. For ordered queues, a dead-lettered message parks its key: later messages for that key are held until the operator redrives or explicitly skips — because skipping silently is how you corrupt an account's state."

Deep dive 4 — Multi-tenancy (4 min).

"Per-tenant quotas at the front end — bytes/s and requests/s for produce and consume — enforced with local token buckets synced every second. Storage quotas per queue cap the backlog: when a tenant hits it, they get 429s, not the cluster. Critical-tier tenants get placement in a separate cell — separate brokers, separate metadata group — so a noisy best-effort tenant can't fill their disks."

Phase 6: Wrap-Up (2–3 minutes)#

"To summarize the contract: an ack means two AZs have it; a delivery means you hold a lease and must be idempotent; a failure means five attempts over about an hour, then a DLQ your team owns. What I didn't cover: cross-region replication — I'd do async mirroring with consumer-offset translation and accept a seconds-scale RPO — and schema governance for payloads. What I'd build next: per-key parking for ordered queues and a self-service redrive UI, because the DLQ is where the human cost of this system lives."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Almost every large design — notifications, payments, feeds, ad aggregation, search indexing — has a queue in the middle. Interviewers use the queue question to test whether a candidate understands the guarantees they're assuming every time they draw that arrow. A Senior engineer has used queues heavily; a Staff engineer has been paged because one silently dropped messages, redelivered 40,000 of them in a storm, or let a single malformed payload stall a partition for six hours. The question separates users of queues from owners of them.

It's also an ownership question in disguise. A queue sits exactly on a team boundary: the producer team, the queue team and the consumer team each believe someone else is responsible for the message that fails. The Staff answer draws those boundaries explicitly.

1.2 The L5 vs L6 Contrast — Visual#

The Senior path is a sequence of component choices. The Staff path is a sequence of promises, each of which has a cost and a failure mode.

1.3 The Staff Question That Cuts Through Everything#

"When a consumer receives this message, what has it been promised — and what has it promised back?"

The answer forces every decision into the open: whether the message can arrive twice (yes), whether it can arrive out of order (depends on the key), how long the consumer has before someone else gets it (the lease), and what the consumer must guarantee (idempotent effect, ack after side effect, heartbeat if slow). A candidate who can answer that question crisply has designed the queue; the rest is capacity planning.

🎯 Staff Move: "Let me write down the consumer contract before the architecture: you may see a message more than once, you have 60 seconds unless you heartbeat, you must ack after your side effect commits, and after five failures it goes to your DLQ. Everything I build is there to make that contract true."

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Work queue. The unit of value is a completed job. Producers don't care which consumer does the work or exactly when; they care that it's done once, eventually, and that failures surface. The design centers on per-message acknowledgement, leases, retries and the DLQ. Ordering is usually unnecessary and, where it's needed, is per entity. Messages have no value after they're processed — retention exists only to survive consumer outages.

Event log. The unit of value is the history. Many consumer groups — search indexing, analytics, fraud, notifications — read the same events independently and at their own pace; a new consumer may need to replay last week. The design centers on partitioning, per-key order, retention or compaction, and schema contracts. Per-message ack is a non-goal; an offset per partition per group is enough. This is the Kafka sweet spot and the backbone of stream processing.

Ephemeral fan-out. The unit of value is now. A typing indicator from three seconds ago is noise. The design centers on low latency and dropping data for slow subscribers instead of buffering it. Persistence is a liability: it adds latency and creates backlogs nobody wants. See Real-time WebSockets for where this lives in practice.

2.2 When NOT to Use a Distributed Message Queue#

🎯 Staff Move: "Before adding a queue, I'd ask whether anyone needs the work done later rather than now. If the caller blocks on the result, a queue is just a slower, less observable RPC."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

Each fault line below is a place where two competent engineers can disagree. The Staff contribution is not the "right" answer — it's naming who pays for each option and committing to a default with a stated reason to deviate.

3.1 Fault Line 1: Ack Durability vs Producer Latency#

The producer ack is the moment ownership transfers. Before it, losing the message is the producer's problem (it will retry). After it, losing the message is the queue's fault — and nobody will notice until a customer does.

The Staff default: quorum ack, min_in_sync = 2 of RF 3, replicas spread across three AZs, with produce rejected when fewer than two replicas are in sync. Periodic fsync every 100–500 ms for the critical tier.

When to deviate: metrics, logs and clickstream where 0.01% loss is cheaper than 5 ms per call and 2× bandwidth — offer an ephemeral tier with leader ack and RF=2. Make it a named tier with a documented loss bound, never a per-producer config flag nobody reviews.

🎯 Staff Move: "Replication factor is how many copies eventually exist. The ack policy is how many exist when I tell the producer 'you can forget this.' I'll specify the second — two copies in two AZs — and I'll reject writes rather than ack with one."

3.2 Fault Line 2: Ordering vs Parallelism#

Ordering and parallelism are the same knob. Any message that must wait for another is a message that can't be processed concurrently. Every ordering guarantee also defines a blast radius for poison messages: everything in the same ordering scope waits behind it.

The Staff default: no ordering unless requested; when requested, per key with one in-flight message per key and key-parking on poison. The dispatcher tracks in-flight keys per partition; a key with a dead-lettered message is held until redrive or explicit skip.

When to deviate: if consumers can tolerate reordering by carrying a version or timestamp (last-writer-wins on entity_version), drop ordering entirely — it's cheaper and removes head-of-line blocking. If a tenant genuinely needs total order (a ledger posting stream), give them a dedicated single-partition queue and document the throughput ceiling in their SLO.

🎯 Staff Move: "I want ordering to be a purchase, not a default. If you need it, you get it per key, one in flight per key, and a poisoned key parks itself — I'd rather delay one account than every account that hashes next to it."

3.3 Fault Line 3: Redelivery Speed vs Duplicate Work#

The lease is a bet on how long processing takes. Bet too short and slow-but-healthy consumers get their messages yanked and redelivered to someone else — duplicate work, duplicate side effects, and sometimes a feedback loop. Bet too long and a crashed consumer's messages sit invisible until the lease expires.

The Staff default: 60 s lease, heartbeat every lease/3, maximum total lease of 1 h (configurable per queue up to 12 h), and stale-receipt acks rejected. A worker that can't finish in the max lease should be a workflow, not a message.

When to deviate: for very short jobs (p99 < 1 s) at high volume, a 10–15 s lease without heartbeats reduces dispatcher load. For batch jobs that legitimately take hours, move them to a job scheduler with checkpointing.

🎯 Staff Move: "The lease should reflect how fast I want to notice a dead consumer, not how long the slowest job takes. Heartbeats decouple the two: 60 seconds to detect a crash, and a job can run as long as it keeps proving it's alive — up to an hour."

3.4 Fault Line 4: Retry Forever vs Dead-Letter#

Every failed message is either retried or set aside. Retrying requires no human but can loop forever; setting aside bounds load but requires an owner. The question is never "should we have a DLQ" — it's who looks at it and how fast.

The Staff default: five attempts with backoff (10 s, 1 m, 10 m, 1 h), classify errors where possible (a 400 Bad Request from the handler goes straight to DLQ — retrying a deterministic failure is waste), DLQ owned by the consuming team, alert on age, retention 14 days in the DLQ vs 4–7 in the primary.

When to deviate: if the downstream dependency has a known long outage window (a partner API with maintenance windows), extend the retry budget to cover it rather than dead-lettering thousands of good messages; the tell is a DLQ where >90% of messages redrive successfully without code changes.

🎯 Staff Move: "A DLQ is a queue with a pager attached, or it's a slower way of deleting data. I'll make the consuming team its owner, alert on the age of the oldest message, and give them a redrive tool that's rate-limited so fixing a bug doesn't create an incident."

3.5 Fault Line 5: Shared Multi-Tenant Cluster vs Dedicated Queues#

The Staff default: a shared cluster per cell with per-tenant quotas on produce bytes/s, consume bytes/s and stored bytes; critical-tier tenants placed in a dedicated cell; best-effort tenants share. Start with one cell; split when a cell exceeds ~30% of org traffic or any tenant exceeds ~20% of a cell.

When to deviate: a single tenant that is 50%+ of total traffic (often the analytics firehose) should get its own cluster — not for isolation from others, but so others are isolated from it.

🎯 Staff Move: "Quotas protect capacity; cells protect against failure. A tenant that sends a 900 KB message with a pathological key can't be stopped by a rate limit — only by putting them somewhere their blast radius is bounded."

4. Failure Modes & Operational Reality#

4.1 The Lease-Expiry Redelivery Storm#

A downstream dependency slows down; handlers that took 5 s now take 70 s; the 60 s lease expires on every in-flight message; each is redelivered to another worker; the dependency's load doubles; handlers slow further.

t=0:        Downstream payments API p99 rises from 4s to 70s (its DB failover).
t=+60s:     Leases on ~12,000 in-flight messages expire. All redelivered.
            Workers now run 2 copies of each slow call. Downstream QPS ×2.
t=+2min:    Downstream p99 → 120s. Second wave of expiries. In-flight ×3.
            queue.redelivery_rate: 0.1% → 64%. ack.stale_receipt_rate spikes.
t=+3min:    Page: queue.redelivery_rate > 10% for 2m. Also downstream 5xx > 20%.
t=+5min:    On-call pauses consumption on the queue (dispatcher flag), not the producers.
t=+12min:   Downstream recovers. Consumers resumed at 25% concurrency, ramp 25%/2min.
t=+30min:   Backlog of 480K drained. 3,100 duplicate side effects — all deduplicated
            by consumer idempotency table except 41 from a handler without it.

Detection: queue.redelivery_rate, ack.stale_receipt_rate, handler.duration_p99 vs lease_seconds, downstream error rate.

Mitigation: heartbeats so live handlers keep their leases; a per-queue concurrency cap that the consumer SDK enforces; client-side circuit breaker on the dependency so handlers fail fast and nack with backoff instead of hanging.

Prevention: alert when handler.duration_p99 > 0.5 × lease_seconds; make heartbeat the SDK default; idempotency conformance test for every handler on a critical-tier queue.

Owner: consuming team (handler and dependency behavior); platform team owns the redelivery-rate alert and the pause control.

4.2 Poison Message Stalls an Ordered Partition#

t=0:        A producer deploy emits an event with a new enum value. Consumer v41
            throws on parse. Queue is ordered per account, partition-blocking.
t=+1s:      Partition 23 (≈ 9,000 accounts) stops advancing. Retries: 10s, 1m, 10m...
t=+5min:    queue.oldest_unacked_age on partition 23: 5 min. Other partitions fine.
            Aggregate lag looks normal — 1 of 64 partitions is stuck.
t=+40min:   Customers on partition 23 report stale balances. Support escalates.
t=+50min:   On-call finds the stuck offset via per-partition age dashboard.
t=+55min:   Manual skip of the message (no key-parking) → account 77812 state now
            inconsistent: later events applied without the skipped one.
t=+3h:      Consumer v42 with the new enum value deployed. Skipped message replayed by hand.

Detection: queue.oldest_unacked_age per partition (max, not average); consumer.parse_error_rate; partition.commit_stalled_seconds.

Mitigation: key-parking — the poisoned key's messages move to a hold queue; the partition continues. Redrive after fix preserves order for the key.

Prevention: schema registry with compatibility checks at produce time (new enum values are a breaking change for strict parsers); consumers treat unknown enum values as a handled case; per-partition age alert at 5 min for ordered queues.

Owner: producing team owns the schema change; consuming team owns the parser; platform owns key-parking and the per-partition alert. The post-mortem question is why a breaking schema change shipped without a compatibility check — that's the producer's gate.

4.3 Backlog Outlives Retention — The Silent Expiry#

t=0 (Fri 18:00): Consumer team's deploy breaks auth to its database. Handlers nack everything.
                 Retries exhaust → DLQ. DLQ alert routes to a Slack channel, not a pager.
t=+2h:           DLQ depth: 1.1M. Primary lag growing 40K/min (nacks are slow).
t=+60h (Mon):    Team notices. Fixes auth. Redrives DLQ.
t=+60h:          Primary queue retention: 4 days. OK. DLQ retention: 4 days (copied default).
t=+96h (Tue):    Oldest messages in the primary expire before consumers catch up:
                 consumers drain at 3× produce rate, but 60h of backlog needs ~30h.
                 ~210K messages expire unprocessed. No error, no log line — just gone.

Detection: queue.oldest_unacked_age approaching retention × 0.5; queue.expired_unprocessed_count (must exist — many systems don't emit it); drain ETA = backlog ÷ (consume rate − produce rate).

Mitigation: extend retention on the live queue (a config change — make sure it's online); add consumer capacity if the downstream can take it; prioritize oldest-first draining.

Prevention: retention ≥ 3× the longest plausible outage plus drain time; DLQ retention > primary; expired_unprocessed_count > 0 pages the platform on-call for critical-tier queues; DLQ alerts route to pagers for critical tier.

Owner: consuming team for the outage; platform team for retention defaults and the expiry metric. A message that expires unprocessed is a data-loss incident, not a consumer-lag ticket.

4.4 Acked Messages Lost After Broker Failure#

t=0:        Storage node B1 (leader for 140 partitions) has a degrading disk.
            Followers fall behind; in-sync set for 31 partitions shrinks to {B1}.
            Cluster configured min_in_sync=1 "for availability" during a migration.
t=+10min:   Producers still get acks for those 31 partitions — single copy.
t=+14min:   B1's disk fails hard. Controller elects lagging followers (unclean election
            enabled to avoid unavailability). ~38 seconds of writes on 31 partitions gone.
t=+14min:   Consumers see offsets jump backward then forward; ~1.9M acked messages missing.
t=+3h:      A downstream team's reconciliation flags missing orders. Incident declared.

Detection: partition.isr_size < min_in_sync_target (should page), cluster.unclean_elections_total, consumer-side gap detection on producer sequence numbers.

Mitigation: none after the fact, beyond replaying from producers' outboxes where they exist. That's the point: the only fix is before.

Prevention: min_in_sync = 2 is non-negotiable on the critical tier and requires a change review to lower; unclean leader election disabled; under-replication alert at 2 min; producers on critical flows use the outbox pattern so the source of truth can replay.

Owner: platform team. The deeper owner is whoever approved min_in_sync=1 "temporarily" — durability config changes need the same review as schema migrations.

4.5 Consumer Group Rebalance Storm#

t=0:        Consumer fleet of 200 pods autoscales to 320 under load.
t=+0–90s:   Each join triggers a group rebalance. Eager rebalancing revokes all
            partitions from all members on every change. 120 joins → near-continuous
            stop-the-world. Throughput: 180K msg/s → 15K msg/s.
t=+3min:    Lag climbs, autoscaler adds more pods → more rebalances.
t=+6min:    Page: consumer.throughput dropped 80%, group.rebalances_per_min > 20.
t=+8min:    On-call freezes the autoscaler. Group stabilizes in ~45s.

Detection: group.rebalances_per_min, group.partitions_unassigned_seconds, throughput vs consumer count (more pods, less throughput = rebalance problem).

Mitigation: incremental/cooperative rebalancing so only moved partitions pause; static membership so restarts don't trigger rebalances; autoscaling in steps with a cool-down ≥ 2× rebalance time.

Prevention: for work queues, the lease-based dispatcher sidesteps group rebalancing entirely — consumers are stateless pollers and adding one costs nothing. This is a strong argument for the lease layer over raw consumer groups when consumer counts are volatile.

Owner: consuming team (autoscaling policy); platform team (rebalance protocol defaults).

4.6 The Noisy Tenant Fills the Disks#

t=0:        Analytics tenant ships a bug: produces each event 40× (retry loop without
            backoff), 1.2 GB/s against a 300 MB/s quota... except storage quota was unset.
t=+20min:   Rate quota caps them at 300 MB/s, but at 7-day retention they're writing
            180 GB/10min into shared brokers. Disk usage 61% → 78%.
t=+50min:   Brokers hit 85% disk. Log cleaner falls behind. Page: broker.disk_used > 85%.
t=+55min:   Platform on-call cuts tenant's retention to 6h and storage quota to 2 TB.

Detection: broker.disk_used_pct, tenant.bytes_stored vs quota, tenant.produce_bytes_rate z-score vs 7-day baseline.

Mitigation: per-queue storage quota; when hit, that tenant's produces get 429 — never cluster-wide back-pressure. Emergency retention reduction is a pre-approved runbook step for best-effort tiers.

Prevention: every queue must declare a storage quota at creation; default derived from rate × retention × 1.5; anomaly alert on tenant produce rate.

Owner: platform team (quotas, isolation); tenant team (the bug). The tenant's quota hit should page them, not the platform.

4.7 Operational Reality Matrix#

🎯 Staff Insight: Notice that half the owners are consuming teams. A queue platform that pages itself for consumer problems burns out its on-call within a quarter. The platform's job is to make the signals precise and route them to the team that can fix them.

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Deep Kafka internals knowledge ≠ queue design. Reciting ISR mechanics, log.segment.bytes and the KRaft migration is impressive and orthogonal. The question is what the consumer is promised.
• Drawing many retry topics ≠ owning failure. A 6-tier retry topology with no owner for the last tier is an elaborate way to delete data later.
• Naming every broker ≠ judgment. A comparison table of RabbitMQ, Pulsar, SQS and Kafka is useful only after the contract is defined.
• High throughput numbers ≠ correct sizing. "Kafka does millions per second" says nothing about whether one consumer can keep up with one partition.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Segment file format, index files, zero-copy sendfile. One sentence and a link to the Kafka guide.
• Leader election algorithm details. "Raft in the metadata group" suffices.
• Wire protocol and serialization format. Mention schema registry for compatibility; skip Avro vs Protobuf debates.
• Compression codec tuning. "Producer-side batch compression, ~3–5× on JSON" is enough.
• Exact JVM/GC tuning of brokers. Operations detail with no design signal.

6.4 Follow-Up Questions to Expect#

1. "What happens if the dispatcher holding lease state crashes?" — Rebuild from the compacted ack log; in-flight leases are treated as expired; duplicates possible, covered by idempotency.
2. "How do you implement delayed delivery?" — Delay tiers (fixed-delay queues) for retries; a time-indexed store for arbitrary delays beyond ~15 min.
3. "How do consumers dedupe without an unbounded table?" — Dedup window bounded by retention + max redelivery horizon; TTL the table at ~2× retention.
4. "What if a producer sends to the wrong partition after you add partitions?" — For keyed queues, partition count changes require a migration: new queue, dual-write, drain, cut over.
5. "How would you support priority?" — Separate queues per priority with weighted polling in the consumer SDK; not per-message priority inside a partition.
6. "How do you bill tenants?" — Stored GB-hours + produced and consumed GB + request count; replication factor multiplies stored cost.
7. "What would you change for a fan-out workload with 10,000 subscribers?" — Different system: drop per-message ack, use a log with many cheap readers or in-memory pub/sub.

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design a distributed message queue."

Drill 2: The Ack#

Prompt: "Your producer got an ack. The leader's host dies 30 ms later. Is the message safe?"

Drill 3: The Slow Consumer#

Prompt: "Most jobs take 200 ms, but some take 8 minutes. Consumers crash a few times a day. Pick a visibility timeout."

Drill 4: Make It Concrete — Sizing#

Prompt: "200K msg/s peak at 2 KB. Size it."

Drill 5: The Dependency Goes Down#

Prompt: "Consumers write to a database that's now at 95% CPU. Lag is growing fast. The team wants to double the consumer count."

Drill 6: The Hot Key#

Prompt: "One tenant's queue is ordered by merchant_id. A single merchant is 30% of traffic. Their partition lags by hours."

Drill 7: Multi-Tenant Isolation#

Prompt: "300 teams share the cluster. One team's bug just tripled their produce rate. How did your design protect everyone else?"

Drill 8: Build vs Buy#

Prompt: "Should we build this, run open-source Kafka, or use a managed cloud queue?"

Drill 9: Changing a Keyed Queue Without an Outage#

Prompt: "A keyed, ordered queue has 32 partitions and needs 128. How do you do it live?"

Drill 10: Multi-Region#

Prompt: "We're adding an EU region. Some queues must stay in the EU; others need to survive a full region loss."

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Incident — Black Friday Fulfillment Backlog#

Context: On Black Friday, produce rate on the fulfillment.jobs queue hits 9× normal. Lag reaches 2.4M messages; the oldest is 38 minutes old. The fulfillment team has already scaled consumers from 80 to 400 pods, and lag is growing faster. The warehouse-management database behind the consumers is at 98% CPU. The on-call escalates to you.

Questions to Surface First:

• What's the consumption rate now vs at 80 pods? (If it went down, consumers are the problem.)
• What's the redelivery rate? Are leases expiring because handlers slowed?
• What's the oldest message's age vs retention and vs the business deadline (orders must ship same day if placed before 14:00)?
• Is all work equal, or are some jobs (express shipping) more urgent?

Typical L5 Approach: Keeps adding consumers or asks the DBA to scale the database vertically; considers raising the visibility timeout to cut duplicates.

Staff Approach: Recognizes the death spiral: 400 pods overload the database, handlers slow past the 60 s lease, redelivery doubles load. Cuts concurrency back to the level the database sustained (~120 pods), confirms heartbeats are on, and splits express-shipping jobs into a priority queue so the business deadline is met for the work that has one.

Principal Approach: Treats "autoscale consumers on lag" as an org-wide anti-pattern. Proposes that consumer autoscaling policies be bounded by a downstream-capacity signal and that peak-season readiness reviews include a load test of the queue and its consumers' dependencies at 10×.

Key Takeaway: "Backlog is the queue doing its job. Lag is only an incident when its age threatens retention or a business deadline — and adding consumers to a saturated dependency turns lag into an outage."

What clears the Staff bar:

• Identifies lease-expiry amplification as the mechanism making it worse
• Reduces concurrency instead of increasing it, and proves it with consumption rate
• Separates deadline-bound work into its own queue

Deep Dive 2: Silent Failure — The DLQ Nobody Owned#

Context: A product manager asks why ~4% of new users over the past two weeks never received a welcome email. No alert fired. You find the email.welcome DLQ holds 380,000 messages, oldest 13 days old. DLQ retention is 14 days.

Questions to Surface First:

• Who owns this DLQ? Where does its alert route — pager, ticket, or nowhere?
• What's the failure reason distribution? One bug or many?
• When does the oldest message expire — tomorrow?
• Are the messages still valid to process (is a 13-day-old welcome email harmful)?

Typical L5 Approach: Redrives the entire DLQ immediately; 380K emails go out in 10 minutes, the email provider throttles the account, and some users get welcome emails two weeks late.

Staff Approach: First extends DLQ retention to stop expiry tomorrow. Samples failures: 99% are a template-render error introduced by a deploy 13 days ago. Gets product sign-off on what to do with stale messages (welcome emails older than 3 days → send a different "getting started" email or drop). Fixes the template, redrives at a rate the email provider tolerates.

Principal Approach: Asks how many other DLQs are in the same state. Commissions a one-time audit of every DLQ in the org, and makes "DLQ has an owner and a paging alert" a precondition for creating any queue in the critical or standard tier.

Key Takeaway: "A DLQ without a paging owner is a 14-day delay on deleting data. The decision about a stale message is a product decision, not an infrastructure one."

What clears the Staff bar:

• Stops the bleeding (expiry) before investigating
• Involves product on what to do with stale messages instead of blindly redriving
• Redrives at a rate the downstream can absorb

Deep Dive 3: Large-Customer Onboarding — The Ledger Team Wants Strict Order#

Context: The ledger team wants to move from a database-polling job system onto the platform. Requirements: 60K msg/s, strict order per account, zero loss, and "exactly-once." Their largest account (an internal treasury account) is 8% of traffic. They want go-live in six weeks.

Questions to Surface First:

• What does "exactly-once" mean for them — no double posting? Their ledger is the effect; can postings be idempotent by posting_id?
• Is order needed per account, or per account-and-currency? What breaks if two postings for one account arrive reversed?
• How fast is a single posting for the treasury account? 8% of 60K = 4.8K/s on one key.
• What's their tolerated end-to-end latency, and what's their outage tolerance?

Typical L5 Approach: Gives them a single-partition queue for strict order and enables broker transactions for exactly-once.

Staff Approach: Converts "exactly-once" into a unique posting_id constraint in the ledger's transaction. Discovers the treasury account needs ~4.8K postings/s serially — a ~0.2 ms budget per posting, impossible with one transaction each. Proposes per-key batching in the consumer: consecutive messages for the same account committed in one ledger transaction (up to 200), preserving order and dropping per-posting overhead. Places them in the critical-tier cell.

Principal Approach: Notices this is the platform's first critical financial tenant and the beginning of a tier that needs its own durability standard, audit trail on redrive and skip operations, and change controls. Uses the onboarding to define that tier, rather than special-casing one team.

Key Takeaway: "Exactly-once is a property of the consumer's transaction. Strict order on a hot key is a throughput ceiling you must measure on day one — batching per key is how you raise it without breaking order."

What clears the Staff bar:

• Translates "exactly-once" into a concrete idempotency mechanism
• Computes the serial budget for the hot key and finds it infeasible before launch
• Uses shadow consumption and an explicit cutover criterion

Deep Dive 4: Post-Mortem — 1.9M Acked Messages Lost#

Context: You're asked to lead the post-mortem for incident 4.4: a storage node failure lost ~38 seconds of acked writes on 31 partitions. Root cause, per the initial write-up: "disk failure." Leadership wants to know whether it can happen again.

Questions to Surface First:

• Why was min_in_sync set to 1? Who changed it, when, and was it reviewed?
• Why was unclean leader election enabled?
• How long were those partitions under-replicated before the failure? Did an alert fire?
• Which consumers or producers could detect the gap? Can any data be recovered from producer outboxes?

Typical L5 Approach: Root cause: disk failure. Action items: replace the disk vendor, add disk SMART monitoring.

Staff Approach: Disk failure is the trigger, not the root cause — disks fail weekly in a fleet this size. Root cause: a durability config was lowered during a migration three weeks earlier and never restored, unclean election was enabled, and the under-replication alert was a dashboard, not a page. Actions: restore and lock config, page on under-replication, recover what's possible from producers' outboxes.

Principal Approach: Classifies durability config as a safety control. Puts it under policy-as-code with drift detection, requires a time-boxed exception with an owner and expiry for any reduction, and adds a monthly automated audit that compares live config to the tier standard.

Key Takeaway: "The disk was the trigger. The root cause was a durability setting that could be lowered without review and never came back. Safety config needs the same rigor as schema migrations."

What clears the Staff bar:

• Refuses "hardware failure" as a root cause
• Finds the config drift and the missing page
• Recovers data via producer outboxes and owns notifying tenants

Deep Dive 5: Multi-Region Expansion — EU Launch With Residency#

Context: The company is launching in the EU in two quarters. Legal requires EU customer data to stay in the EU. Two critical-tier tenants (payments and orders) also want to survive a full region outage. Today everything runs in one US region.

Questions to Surface First:

• Which queues carry EU personal data? Does the residency rule cover message payloads only, or metadata (keys, headers) too?
• For DR: what RPO and RTO do payments and orders need? Is seconds of loss acceptable or must it be zero?
• Do any flows legitimately cross regions (EU orders triggering US fulfillment)?
• Who decides when to fail over a region — automation or a human?

Typical L5 Approach: Deploys a second cluster in the EU and mirrors everything bidirectionally.

Staff Approach: Splits residency from DR. EU tenants get queues in an EU cell; the front end routes by tenant home region. Cross-region flows go through an explicit, audited bridge that strips or tokenizes personal data. For DR, payments and orders mirror asynchronously to a standby region within the same jurisdiction, with ack state mirrored for offset translation; RPO measured at ~2–5 s.

Principal Approach: Uses the launch to introduce a region-and-tier placement model: every queue declares data classification and DR tier at creation, and the platform enforces placement. Calculates the mirroring bill and limits DR mirroring to queues whose owners accept the cost on their budget.

Key Takeaway: "Residency and disaster recovery are two different requirements with two different mechanisms. Mirror only what has an owner willing to pay for it."

What clears the Staff bar:

• Separates residency from DR and solves each explicitly
• Rejects synchronous cross-region acks with a latency cost
• Handles offset translation and fencing as first-class failover problems

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Distinguish work queue, event log and ephemeral fan-out, and explain why one design can't serve all three well
• Define the producer ack in terms of replicas and failure domains, and defend rejecting writes below the in-sync minimum
• Explain why delivery is at-least-once and how consumers achieve exactly-once effect
• Design leases with heartbeats and receipt generations, and size the lease from crash-detection needs
• Bound the blast radius of a poison message with per-key in-flight limits and key-parking
• Set retention from the longest plausible consumer outage plus drain time
• Assign ownership: platform owns contract, tooling and signals; consuming teams own handlers and DLQs
• Size partitions and brokers from throughput, replication and retention — and say which resource binds first
• Explain why repartitioning a keyed queue is a migration, not a config change

The Bar for This Question#

Mid-level (L4): Describes producers, brokers and consumers correctly; picks a well-known product; mentions replication and consumer groups. Can explain what a queue is for. Struggles when asked what happens on consumer crash.

Senior (L5): Designs a reasonable partitioned, replicated system with consumer groups, retries and a DLQ. Knows the terms — ISR, offsets, visibility timeout. Handles the happy path and the obvious failures well. Under pressure, claims exactly-once, defaults to global or partition ordering, and treats the DLQ as a box on the diagram.

Staff+ (L6): Starts from the contract. Defines the ack in failure domains, the consumer's obligations, the lease and its heartbeat, the poison path and its owner. Treats duplicates and backlogs as routine operation, not edge cases. Assigns failures to the teams that can fix them. Uses numbers to make decisions — partition budget, lease length, retention — and knows which decisions are hard to reverse. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Exactly-Once Delivery" Is the Wrong Requirement#

The Staff position: Ask for at-least-once delivery and exactly-once effect. Put dedup where the side effect commits.

Why this matters in interviews: "Exactly-once" is the single most common overclaim in queue interviews. Correcting it — politely and with the mechanism — is a strong Staff-level signal.

10.2 Most Teams Asking for Ordering Don't Need It#

The Staff position: Default to unordered. Grant per-key order when a named invariant requires it, and make the requester own the throughput ceiling.

Why this matters in interviews: Pushing back on a requirement with a cheaper mechanism that preserves the invariant is exactly what Staff engineers do in design reviews.

10.3 Your Queue's Biggest Risk Is Its Consumers, Not Its Brokers#

The Staff position: Spend the platform's engineering budget on the consumer SDK — heartbeats, idempotency, concurrency limits, error classification — before the next broker optimization.

Why this matters in interviews: It shifts the conversation to where real incidents happen, which interviewers who've been on call recognize immediately.

10.4 A Database Table Is the Right Queue More Often Than You Think#

The Staff position: Below ~1K jobs/s, when the job must be consistent with business data, a Postgres table is a fine queue. Graduate to a log when throughput, fan-out or retention needs outgrow it.

Why this matters in interviews: Knowing when not to introduce infrastructure is a judgment signal; interviewers notice candidates who reach for a cluster to move 50 jobs a second.

10.5 DLQ Retention Should Be Longer Than Primary Retention#

The Staff position: DLQ retention ≥ 2× primary retention, with an age alert at 1 h. Expiry from a DLQ should be an incident.

Why this matters in interviews: It's a small, concrete, defensible choice that shows you've watched a DLQ expire.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

The Staff engineer designs a queue with a correct contract. The Principal engineer notices that the org runs five queueing systems — a self-run Kafka for events, SQS for some teams' jobs, a RabbitMQ cluster from an acquisition, a Redis-based job runner, and a Postgres jobs table in the monolith — each with its own delivery semantics, its own DLQ convention and its own on-call. Incidents don't come from any single system's design; they come from the seams: a team that assumes Kafka redelivers like SQS, a DLQ convention that only exists in one of them, five sets of runbooks. The L7 problem is asynchronous-messaging governance: which systems the org keeps, what contract all of them honor, and who is accountable for messages between teams.

The Org-Level Fault Line#

One messaging platform vs a curated set of systems vs team choice.

🧭 Principal Move: "Three systems, one contract. A log for events, a work queue for jobs, pub/sub for ephemeral fan-out — and every one of them honors the same consumer SDK, DLQ ownership rule and durability tiers. The acquisition's RabbitMQ and the Redis job runner get an 18-month retirement plan."

Cost Model#

Assumptions: cloud-hosted, three AZs, RF=3, average message 2 KB, 7-day retention on critical tier and 2 days on best-effort, tiered storage for segments older than 24 h, fully loaded engineer ~$250K/year, cross-AZ transfer billed at typical cloud rates.

The pricing insight: at enterprise scale, best-effort tenants with 7-day retention they don't need are often 30–50% of the storage bill. A retention-tier policy with chargeback typically pays for two platform engineers in its first quarter. Conversely, cross-region mirroring for every queue "just in case" can double the transfer bill; mirroring only the ~5–10% of queues with a real DR requirement is the single largest lever.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-MSG-001: Asynchronous Messaging Standard
Status: Approved   Owners: Messaging Platform + Production Engineering

Scope
  Every service that produces or consumes messages across a team boundary,
  on any approved system (event log, work queue, pub/sub).

MUST
  1. Use an approved system: the event log, the work queue, or pub/sub.
     New use of other brokers requires an exception.
  2. Declare tier at creation: ephemeral, standard or critical. Tier sets
     ack policy, retention, DR eligibility and alert routing.
  3. Carry a stable message_id in the platform envelope. Consumers of standard
     and critical queues MUST be idempotent on message_id and pass the
     redelivery conformance test.
  4. Ack only after the side effect commits.
  5. Every standard and critical queue has a DLQ with a named owning team,
     a paging alert on oldest-message age (critical: 15 min, standard: 1 h),
     and DLQ retention at least 2x primary retention.
  6. Messages expired unprocessed on critical queues are incidents.
  7. Durability settings below the tier standard require a time-boxed
     exception with an owner and expiry.

SHOULD
  1. Prefer unordered queues; per-key ordering requires a stated invariant
     and a key cardinality estimate.
  2. Payloads over 256 KB use claim-check via blob storage.
  3. Use the platform consumer SDK (heartbeats, concurrency limits, error
     classification) rather than raw broker clients.

Exceptions
  Filed with Messaging Platform; reviewed within 5 business days; time-boxed
  to 2 quarters; durability exceptions need Production Engineering sign-off.

Success metrics
  - Acked-then-lost messages on critical tier: 0
  - Messages expired unprocessed (critical): 0; (standard): < 1 per million
  - DLQs with no owner: 0
  - Median DLQ resolution time: < 24 h
  - Legacy brokers in production: 0 by end of year 2

What I'd Tell the VP#

"We run five different messaging systems, and most of our async incidents happen in the gaps between them — a team assuming one system behaves like another, or failed messages piling up somewhere nobody watches. I'm proposing we consolidate to three systems that share one contract: what a 'sent' message guarantees, who owns failures, and how long we keep data. It's about four engineers for a year, retires two legacy clusters, and should cut async-related incidents by roughly half. We'll also introduce retention tiers and chargeback, which we expect to reduce the messaging infrastructure bill by 25–35%. The main risk is migration effort for teams on the legacy systems; we'll provide tooling and an 18-month window."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "I'd build a great work queue with leases and DLQs" — correct, but ignores the four other systems teams already use and the seams between them.
• "Each team owns its DLQ" — names an owner but not the enforcement: creation-time checks, paging routes and an org-wide expiry metric.
• "We'll replicate across regions for resilience" — no classification of which queues need it, no cost, no failover decision owner.

Appendices

Appendix A: Mechanics in Depth#

A.1 Lease-Based Delivery on Top of a Log#

The dispatcher turns an offset-based log into a per-message work queue. For each partition it owns, it keeps:

• committed_offset — every message below this is acked
• in_flight — map of offset → (lease_expiry, generation, consumer_id)
• acked_above_commit — sparse set of acked offsets above the commit point
• attempts — offset → attempt count (only for messages that have failed)
• busy_keys — keys with a message in flight (for ordered queues)

on receive(consumer, max_n, lease_s):
    batch = []
    for offset in scan(from = committed_offset):
        if offset in in_flight or offset in acked_above_commit: continue
        msg = log.read(offset)
        if queue.ordered and (msg.key in busy_keys or msg.key in parked_keys): continue
        gen = next_generation(offset)
        in_flight[offset] = (now + lease_s, gen, consumer)
        if queue.ordered: busy_keys.add(msg.key)
        batch.append(msg with receipt = (partition, offset, gen))
        if len(batch) == max_n: break
    persist_lease_events(batch)              # to compacted ack log
    return batch

on ack(receipt):
    (p, offset, gen) = receipt
    if in_flight.get(offset).gen != gen: reject STALE_RECEIPT
    remove in_flight[offset]; acked_above_commit.add(offset); release key
    advance committed_offset while committed_offset in acked_above_commit
    persist_ack(offset)

on lease_expired(offset) or nack(offset, retry_after):
    attempts[offset] += 1
    if attempts[offset] > max_attempts:
        dlq.produce(msg, history); mark acked; if ordered: park(msg.key)
    else:
        delay_tier(attempts[offset]).produce(msg)  # 10s, 1m, 10m, 1h
        mark acked in this partition               # the retry copy carries the obligation

Why it's right: the log stays append-only and sequential; per-message state is only kept for the window between commit point and head (usually seconds of messages), so memory is bounded by in-flight count, not backlog size. Why it can go wrong: a single message stuck far below the head keeps acked_above_commit growing — cap the gap (e.g., 100K offsets) and force the stuck message to the retry path when exceeded.

A.2 Ordered Retry Without Head-of-Line Blocking#

For ordered queues, a retried message must not be overtaken by a later message for the same key. The dispatcher keeps the key in busy_keys until the retry resolves; later messages for that key are skipped during scans (they stay unacked in the log). If the retry exhausts attempts, the key is parked: its later messages are copied to a per-key hold area in order and acked in the main partition, so the partition's commit point advances. Redrive replays the hold area for that key, in order, before releasing the key.

A.3 Why Not Per-Message State Stored in a Database?#

A per-message row (status, lease_expiry) in a database is the simplest work queue, and fine at low rates. At 200K msg/s it means ~600K writes/s (insert, lease, delete) with index churn, and every receive is a range scan with locking. The log + lease layer keeps the per-message write path append-only and keeps mutable state proportional to in-flight messages.

Appendix B: Message Envelope and Identity#

envelope {
  message_id:     string   # producer-supplied UUIDv7 or derived from business key
  queue:          string
  key:            string?  # ordering and partitioning key, optional
  schema_ref:     string   # registry subject + version
  produced_at:    int64    # producer clock, ms
  enqueued_at:    int64    # broker clock, ms (authoritative for age)
  attempt:        int32    # set by dispatcher on delivery
  trace_context:  string   # propagated for end-to-end tracing
  headers:        map<string,string>
  payload:        bytes    # ≤ 256 KB; otherwise claim-check pointer
}

Identity rules. message_id must be stable across producer retries — generate it before the first send, not inside the retry loop. Derive it from a business key (order_id:event_type:version) when the producer itself may be re-run (batch jobs, replays), so a re-run produces the same IDs and dedup works across runs.

Consumer dedup table.

CREATE TABLE processed_messages (
  message_id   text PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);
-- in the handler's transaction:
INSERT INTO processed_messages (message_id) VALUES ($1)
  ON CONFLICT DO NOTHING RETURNING message_id;   -- no row returned → duplicate, skip effect
-- TTL: delete rows older than 2 × (retention + max retry horizon)

Appendix C: Coordination Mechanisms#

C.1 Producer Idempotence#

The broker assigns each producer session an ID; the producer numbers its batches per partition. The leader rejects a batch whose sequence number it has already appended (duplicate retry) and fails a batch with a gap (lost batch). This makes retries after a lost ack safe within a producer session. It does not cover a producer process restart that re-sends the same business event — that's what a business-derived message_id is for.

C.2 Quick Comparison#

Appendix D: API Contract & Client Behavior#

Appendix E: Observability#

Core metrics (per queue, per partition where noted):

Control plane vs data plane. The data plane (produce, receive, ack) must keep working when the control plane (metadata store, rebalancer, config service) is down: leaders keep serving, dispatchers keep leasing, front ends use cached partition maps. Control-plane outage blocks queue creation, config changes and leader moves — painful, but not an outage for tenants. Test this explicitly.

Debugging the silent failure. Aggregate lag hides single stuck partitions: always chart the max per-partition age, not the average. For "missing message" reports, trace by message_id through produce log → partition offset → lease events → ack or DLQ. If the platform can't answer "where is message X?" in under a minute, the incident will take hours.

Appendix F: Scale Evolution#

Multi-region path: single region, three AZs → async mirroring for DR-tier queues to a standby region → region-homed tenants with residency enforcement → (rarely) active-active per key range with fencing. Most organizations never need the last step.

What You Don't Build on Day One: your own storage engine; cross-region active-active; per-message priority inside a partition; a custom delay scheduler for arbitrary future times (use a scheduler); broker-side transactions for external effects (they don't exist).

Appendix G: Multi-Tenancy, Fairness & Cost#

Fairness at the dispatcher. When many consumers poll a shared partition set, serve receives round-robin across queues with weights, not FIFO across all requests — otherwise a tenant with 500 polling workers starves a tenant with 5.

Quotas.

Cost attribution. Charge on stored GB-hours × replication factor, produced and consumed GB, and mirrored GB. Show tenants their bill monthly; the first bill typically prompts 20–30% of tenants to shorten retention or move to the best-effort tier, which is the point.