Technologies referenced in this case study: PostgreSQL · Redis · DynamoDB · Apache Kafka · Flink & Stream Processing · API Gateways

How to Use This Case Study#

Organized for interview use first, reference second. Distributed Coordination introduces idempotency keys and the outbox as tools; Payment Processing applies them to one PSP integration. This case study goes underneath both: the exact semantics of a key, the record's lifecycle, concurrent duplicates, what to replay, how idempotency survives crossing three service boundaries and a message broker, and how to make it the org's default rather than each team's project.

Without idempotency:
t=0:       User taps "Place order" on a train; request reaches server
t=+1.2s:   Server commits order #A, charges card, enqueues fulfilment
t=+1.3s:   Response lost — tunnel. App shows spinner, then "Network error"
t=+4s:     App auto-retries (3 attempts policy) → server creates order #B, charges again
t=+9s:     User sees "Network error" again, taps "Place order" manually → order #C
t=+2 days: Three parcels arrive. Two chargebacks. Support ticket. App-store review: 1 star.

With an idempotency key per logical operation:
t=0:       App generates key k=0190f3…(UUIDv7) when the user taps, persists it locally
t=+1.2s:   Server: INSERT key k (in_progress) → commit order #A + charge + outbox → key k completed
t=+1.3s:   Response lost
t=+4s:     Retry with SAME key k → server finds completed record → replays order #A response
t=+9s:     User's manual tap reuses pending key k (button bound to the in-flight operation)
Result:    One order, one charge, one parcel. idempotency.replay_total += 2.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the intents and fault lines below.

What This Interview Actually Tests#

Idempotency is not a header question. Everyone says "Idempotency-Key."

This is an unknown-outcome question that tests:

• Whether you treat a timeout as a third outcome — not success, not failure — and design for it
• Whether you can define a key's scope, lifetime and meaning precisely, not just name it
• Whether you know what happens when two copies of the same request are in flight at once
• Whether the guarantee survives crossing service boundaries and a message broker, or quietly stops at the first hop
• Whether you'd make it a platform default so 40 teams don't each get it 80% right

The key insight: Exactly-once is not a delivery guarantee you buy; it is an effect guarantee you assemble — at-least-once delivery, plus a durable record of what has been done, written atomically with the doing. Every gap between "the record" and "the effect" is where duplicates come from.

The L5 → L6 → L7 Contrast — Start Here#

The Staff Positions#

The Four Intents#

🎯 Staff Move: "I'll focus on the first two — a public mutating API whose handler calls downstream services and emits events — because that's where the unknown-outcome problem bites hardest. Then I'll carry the guarantee through the event to the consumer. Pipeline exactly-once is a real but separate design; I'll touch it at the end."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: These are public designs. The tradeoff each one made is the thing to cite.

Stripe — Idempotency Keys as a Public Contract#

Stripe accepts an Idempotency-Key header on POST requests, stores the result of the first request, and returns that same result — status code and body — for subsequent requests with the same key. Its documentation says keys can be pruned after they're at least 24 hours old, that a retry with the same key but different parameters is rejected as an error, and that results are saved only once the endpoint has begun executing — a request that failed validation or collided with a concurrently executing request with the same key does not save a result. Stripe's engineering writing describes the server side as atomic phases with recovery points, so a request interrupted mid-way can be resumed rather than redone.

Staff insight: Stripe's contract has three clauses — same key → same response; different payload → error; concurrent → conflict, nothing saved. Saying all three in an interview is the signal; "we dedupe on the key" says one.

AWS — Client Tokens and "Making Retries Safe"#

Many AWS APIs (EC2 RunInstances, for example) accept a client token. Reusing a token with the same parameters returns the original result; reusing it with different parameters returns an IdempotentParameterMismatch error. The Amazon Builders' Library article on making retries safe with idempotent APIs describes the reasoning: the caller supplies the identifier for the intent, and the service records it atomically with the mutation so retries collapse onto the first execution.

Staff insight: AWS treats the parameter-mismatch error as a first-class part of the API. It is the guard against the most common client bug — reusing a key for a different operation — and it converts a silent wrong answer into a loud one.

Apache Kafka — Idempotent Producer and Transactions#

Kafka's idempotent producer assigns each producer a producer ID and per-partition sequence numbers, so the broker discards duplicates caused by producer retries (enabled by default since Kafka 3.0). Kafka transactions extend this so a consume-transform-produce application can commit its output records and its consumer offsets atomically; downstream consumers in read_committed mode see each result once.

Staff insight: Kafka's exactly-once is real inside Kafka. The moment a consumer writes to Postgres, calls an API or sends an email, the guarantee ends and consumer idempotency begins. Candidates who say "Kafka gives us exactly-once" without that caveat get downleveled.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: The idempotency record, the business row and the outbox row live in one database and commit together — that is the whole correctness argument. External calls (inventory, PSP) cannot join that transaction, so each gets a derived key and the handler records a recovery point before and after them. The gateway enforces the presence of keys but does not retry. Events leave via the outbox at-least-once, so every consumer dedupes. The metrics row is how you find out the guarantee is leaking.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates say "idempotency key" in minute three, draw a Redis box, and spend the next twenty minutes on unrelated scaling. The interviewer wanted the twenty minutes on what the key means, what happens to the second copy, and what happens one hop downstream.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional requirement in one sentence:

"Clients must be able to retry any mutating request after a timeout or error without causing the side effect twice, and the guarantee must hold through our downstream calls and the events we emit."

Then name where duplicates come from and commit:

"Duplicates come from four sources: client retries, our internal retries, broker redelivery, and replays. I'll design for a public POST /orders whose handler reserves inventory, charges a card through a provider, and emits an OrderPlaced event. Clients include mobile apps with offline retry queues — so retries can arrive hours later. Correctness bar: zero duplicate orders, charges or shipments per logical operation."

Commit to the constraint set:

"Keys are generated by the client per logical operation; the key record commits atomically with the order; concurrent duplicates never execute twice; timeouts are unknown outcomes, not failures; and every hop downstream carries a derived key."

🎯 Staff Move: Naming the four duplicate sources up front tells the interviewer you know that the header handles only one of them.

Phase 2: Core Entities & API (1–2 minutes)#

• IdempotencyRecord: (tenant_id, endpoint, key) PK, fingerprint, status (in_progress / completed), recovery_point, locked_until, response_code, response_body, created_at, expires_at
• OutboxEvent: event_id, aggregate_id, type, payload, created_at, published_at
• InboxRecord (consumer side): (consumer, message_id) PK, processed_at

API contract:

POST /v1/orders
Idempotency-Key: 0190f3a2-7c1e-7b3a-9f21-4d8e6c0b5a11     (required; 16–64 chars)
→ 201 Created            first execution, or replay of a completed one (header: Idempotent-Replayed: true)
→ 409 Conflict           same key still in progress   (Retry-After: 1)
→ 422 Unprocessable      same key, different payload  (error: idempotency_key_reused)
→ 400 Bad Request        missing or malformed key on an endpoint that requires one
→ 410 Gone / 422         key older than retention     (error: idempotency_key_expired)

🎯 Staff Move: "The status codes are part of the contract. 409 tells the client 'wait and retry the same key'; 422 tells it 'you have a bug — you reused a key for a different operation.' If those collapse into one generic error, every SDK handles them differently." (The IETF HTTPAPI working group's Idempotency-Key draft uses this same 400/409/422 split.)

Phase 3: High-Level Architecture (≤5 minutes)#

Walk it in 90 seconds:

1. Middleware inserts the key record as in_progress — the unique constraint on (tenant, endpoint, key) is what makes "first one wins" atomic.
2. The handler runs in phases. Local writes commit with a recovery point. External calls carry a key derived from the parent (k:charge), so the provider dedupes our retries.
3. The final phase commits the business state, the outbox event and the completed key record (with the response) in one transaction.
4. A retry finds completed and replays the stored response byte-for-byte.
5. The outbox relay publishes the event at-least-once; consumers dedupe.

🎯 Staff Move: "That's the happy path and the simple retry. The interesting parts are the second copy arriving while the first is still running, the first attempt dying halfway after charging the card, and what 'the same request' even means. Let me take those in order."

Phase 4: Transition to Depth (1 minute)#

"Three places to go deep: the key's semantics — scope, lifetime, fingerprint, and what happens after it expires; the in-flight and crashed-halfway cases — which is where unknown outcomes live; and carrying the guarantee across service and broker boundaries. Which would you like first?"

If the interviewer has no preference, lead with crashed-halfway: it forces the recovery-point design and the unknown-outcome discussion, which is the deepest Staff-level signal.

Phase 5: Deep Dives (25–30 minutes)#

Deep dive A — Key semantics (6–8 min). Client-generated per logical operation, minted before the first attempt and persisted until a terminal outcome. Scope: (tenant, endpoint, key) — never global, or one tenant's key can collide with or probe another's. Fingerprint: SHA-256 of canonicalized method + path + body (minus volatile fields). TTL ≥ max client retry horizon (here 72h for the offline queue). Expired keys: if the key is a UUIDv7, its embedded timestamp lets the server reject a too-old key explicitly instead of re-executing.

Deep dive B — Concurrency and crash recovery (8–10 min). Second copy arrives while first is in_progress → 409 + Retry-After. Lease (locked_until) on the in-progress record. If the first handler crashed, the next retry after lease expiry takes over and resumes from the last recovery point — re-calling the provider with the same derived key, which returns the original charge. Fencing via a version column so a slow original can't commit after a takeover.

Deep dive C — Replay semantics (4–5 min). Replay completed outcomes, including deterministic 4xx (card declined is an outcome). Do not store transient failures before any side effect (validation error, 503 from our own overload) — release the key so a retry can execute. 5xx after a possible side effect is not final: the record stays in_progress with a recovery point, and resolution happens via takeover or a background resolver.

Deep dive D — Across boundaries (6–8 min). Derived keys for every downstream call. Retries at exactly one layer (the SDK), with deadline propagation so downstream services stop work the caller has abandoned. Outbox for events; consumer inbox in the same transaction as the consumer's effect, or versioned upserts. Dedup window ≥ Kafka retention for consumers that can be replayed.

Deep dive E — Platform (3–5 min). Middleware library + schema migration per service (not a central service, because the record must share the service's transaction); gateway enforces key presence; SDKs in every client language; durability tiers.

Phase 6: Wrap-Up (2–3 minutes)#

"To summarize: client-minted keys per operation, scoped and fingerprinted; the key record commits with the effect; concurrent copies get 409; crashed attempts resume from recovery points with derived downstream keys; timeouts are unknown, never failed; events leave via an outbox and consumers dedupe in-transaction. Day one I'd ship the middleware for the five money-moving endpoints and the SDK change. I'd defer the Redis cache, durability tiers and cross-region key routing until replay volume or a second region justifies them. The metric I'd watch from day one is unknown outcomes older than an hour — that's where real duplicates and real losses hide."

🎯 Staff Move: End on the metric that catches leaks. It shows you expect the design to be imperfect and know where to look.

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Idempotency is where distributed-systems theory meets an operational reality you can't avoid: networks drop responses, and the client cannot distinguish "never arrived" from "arrived, executed, response lost." Every write API in every company must answer that question, and most answer it inconsistently — one team with a Redis set, another with a DB table, a third with nothing. The problem rewards candidates who think in invariants ("the record and the effect are the same fact"), in failure timelines ("crash after charge, before commit"), and in contracts ("409 means retry the same key"). It also has an obvious organizational dimension: idempotency is only as strong as the weakest hop in the call chain, which makes it a platform problem whether or not anyone admits it.

1.2 The L5 → L6 → L7 Contrast — Visual#

The L5 path handles the retry that arrives two seconds later. The L6 path handles the retry that arrives during execution, after a crash, 30 hours later, or via a broker. The L7 path makes the L6 path the default for 40 teams.

1.3 The Staff Question That Cuts Through Everything#

"If the process crashes right after the side effect and right before recording it, what happens on retry?"

Every idempotency design either has an answer to this or has a duplicate-effect bug. Same-transaction storage answers it for local effects: the record and the effect commit or roll back together. External effects can't join the transaction, so the answer is a derived key passed to the external system (it dedupes our retry) plus a recovery point (we know which phase we reached). If the external system has no idempotency support, the answer is a pre-call record of intent, a lookup-by-reference after the crash, and reconciliation for the residue. Ask this question about every effect in the handler and the design writes itself.

2. Problem Framing & Intent#

2.1 The Four Intents — Explained#

Intent 1: Client retry safety on a public API. Clients you don't control, on networks you don't control, retrying on schedules you don't control. Mobile SDKs with offline queues can retry 1–3 days later. Browser users double-click and open two tabs. Partners run cron jobs that retry failed batches the next morning. The key must be generated by the client (only the client knows that two requests are "the same operation"), and the server's job is to remember long enough and reject misuse loudly.

Intent 2: Internal call-chain safety. Service A calls B calls C. Each hop has a client library with its own retry policy, and a service mesh may add another. Without derived keys, A's single retry becomes two executions in C. Without a single-layer retry policy, a slow C turns one user request into 27–81 requests during an incident. The design here is propagation: one root key from the client, derived deterministically at every hop (root:step), plus deadlines so downstream stops work the caller already gave up on.

Intent 3: Consumer idempotency. Brokers deliver at-least-once. Kafka redelivers on consumer-group rebalance and after a crash before offset commit; an operator resetting offsets replays days. SQS redelivers after visibility timeout. Webhook senders retry for hours. The consumer must record "I processed message M" atomically with its effect — or make the effect naturally idempotent (versioned upsert). The dedup window must outlast the longest possible redelivery, which for a replayable log is the retention period, not "a few minutes."

Intent 4: Exactly-once pipelines. Aggregations and materializations where each input must contribute exactly once to the output. Kafka transactions and Flink's checkpoint-aligned two-phase-commit sinks give this within participating systems. Everywhere else the answer is deterministic output keys (so replays overwrite rather than append) and idempotent sinks. See Stream Processing and Data Pipeline Patterns.

Committing:

🎯 Staff Move: "These four have different owners and different mechanisms. The client-facing API is the hardest because the client is outside our control and retries can arrive days later. I'll design that, carry it through one downstream call and one event, and treat pipeline exactly-once as a separate conversation."

2.2 When NOT to Build Idempotency Keys#

🎯 Staff Move: "If the operation already has a natural key — the partner's own order reference — I'd make that unique per tenant and skip a separate idempotency table. The best idempotency key is one the domain already has."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: Same-Transaction vs Separate Store#

The tension: The only way to make "the key says done" and "the effect is done" the same fact is to commit them together. That couples the idempotency record to each service's database. A shared fast store (Redis, a central DynamoDB table) is generic and easy to adopt — and has a crash window between effect and record.

Staff default: Same transaction as the effect. Redis only as a cache of completed records (immutable, so caching is safe). For endpoints whose effects are entirely external, the record lives in the service's own DB and recovery points bracket the external calls.

When to deviate: Low-stakes endpoints (likes, follows, preference toggles) where a rare duplicate is harmless — Redis-only is fine and much cheaper to adopt. Make this an explicit tier, not an accident.

🎯 Staff Move: "I'd rather pay 1ms on the write path than explain to finance why every Redis failover creates a few hundred duplicate charges. Redis can cache completed records — those never change — but it can't be the source of truth."

3.2 Fault Line 2: Key Scope & Lifetime#

The tension: Longer retention protects against later retries and costs storage. Narrower scope prevents cross-tenant collisions and probing but needs the server to know the scope at lookup time.

Who generates:

Scope: (tenant_id, endpoint, key). Global scope lets tenant A's key collide with tenant B's (vanishingly rare with UUIDs, common with partner-chosen keys like "order-1"), and lets a malicious tenant probe whether a key exists elsewhere. Endpoint scope prevents a key reused across /orders and /refunds from replaying the wrong resource type.

Lifetime:

The expiry trap: After the record expires, a late retry executes again. Staff fix: make expiry explicit. If keys are UUIDv7 (see ID Generation), the server can read the key's embedded timestamp and reject any key older than the TTL with idempotency_key_expired instead of executing it. Alternatively the SDK refuses to retry past TTL − margin and surfaces the operation as unknown.

Staff default: Client-minted UUIDv7 per logical operation; scope (tenant, endpoint, key); TTL 72h for public APIs with mobile clients (24h if none); expired keys rejected, never re-executed.

3.3 Fault Line 3: Concurrent Duplicates#

The tension: Two copies of the same request arrive 5ms apart (double-click, SDK retry racing a slow original, load balancer retry). Both pass a naive "check key, not found" read.

Staff default: Atomic insert of the in_progress record (unique constraint) before any work. A concurrent duplicate gets 409 Conflict with Retry-After: 1. Optionally block for ≤ 1–2s on a cheap notification (e.g., LISTEN/NOTIFY or a short poll) before returning 409, for better UX on double-clicks. Never hold connections longer.

The crash sub-case: The first copy dies while in_progress. Without a lease, every retry gets 409 forever. With locked_until = now + lease (2–3× handler p99.9, e.g., 30s), the next retry after expiry takes over by atomically updating locked_until and incrementing version, then resumes from the recorded recovery_point. The original, if merely slow, will fail its final commit because its version no longer matches — a fencing token for the handler.

3.4 Fault Line 4: Replay vs Re-evaluate#

The tension: On a repeat request, return the original response (byte-identical) or the current state of the resource?

Which outcomes to store as final:

Staff default: Replay stored responses for completed outcomes with a header (Idempotent-Replayed: true). Release the key only when you can prove no side effect occurred. Everything else stays in_progress until resolved.

🎯 Staff Move: "The dangerous response is the 500 after we called the provider. If I store that 500 as final, the client is told 'failed' for a charge that succeeded. If I release the key, the retry charges again. Neither — the record stays in progress and the retry resumes from the recovery point."

3.5 Fault Line 5: Generic Middleware vs Domain-Native Idempotency#

The tension: A middleware layer gives every endpoint the same guarantees with zero domain thought. Domain-native idempotency — natural keys, conditional writes, state machines — gives stronger, cheaper guarantees but requires every team to think.

Staff default: The middleware owns the external contract (key validation, scope, fingerprint, 409/422, replay). The handler owns correctness of its effects (state machine transitions with conditional writes, derived keys for downstream, outbox for events). A middleware that wraps a handler doing dual writes gives a false sense of safety.

4. Failure Modes & Operational Reality#

Idempotency failures are almost always silent: a duplicate charge, a second shipment, an event counted twice — or, worse, a legitimate second operation silently swallowed as a "duplicate." Nobody is paged; a customer or a reconciliation job finds it. A Staff design makes each class of leak measurable.

4.1 Key Retention Shorter Than the Retry Horizon#

t=0:        Mobile app v5 adds an offline queue: failed POSTs retried on reconnect, up to 7 days
t=0:        Order service idempotency TTL: 24h (copied from a blog post)
t=+3 days:  User on a long-haul trip; 41 queued "place order" requests replay on landing
t=+3 days:  Keys expired → server executes all 41 as new orders
t=+3 days:  Fleet-wide: ~0.3% of mobile orders duplicated during holiday travel week
t=+10 days: Support volume up 18%; refunds $410K; app team and order team each think the other owns it

Detection: idempotency.key_age_at_request_seconds histogram (keys older than TTL on arrival should be ~0); idempotency.expired_key_rejections_total; duplicate-order detector (same user, same cart hash, < 10 min apart, different keys).

Blast radius: Every client whose retry horizon exceeds the server TTL — typically one client platform, all its users.

Mitigation: Raise TTL to 7d for the order endpoint (storage: ~10M orders/day × 2 KB × 7 ≈ 140 GB — affordable). Reject keys older than TTL explicitly using the UUIDv7 timestamp, so the client gets idempotency_key_expired and asks the user rather than silently re-ordering.

Prevention: The TTL and the SDK's max retry horizon are one contract, published together; the SDK enforces retry_horizon ≤ TTL − 1h. A contract test in the mobile CI asserts it.

Owner: API platform owns the published TTL; the mobile team owns the SDK's horizon; both sign the contract. Before the incident there was no owner of the pair — that was the bug.

4.2 Idempotency Store Loses Keys on Failover#

t=0:        Keys stored only in Redis (primary + async replica), SET NX with 24h TTL
t=+0s:      Primary dies; replica promoted; last ~800ms of writes not replicated
t=+0s:      ~2,400 keys for in-flight and just-completed requests lost
t=+1s:      Clients that timed out during the failover retry with the same keys
t=+1s:      Keys not found → handlers re-execute → ~600 duplicate effects
t=+1 week:  Reconciliation flags 600 duplicate charges; refunds issued

Detection: redis.replication_offset_lag at failover time; idempotency.store_failover_total; a post-failover reconciliation job that scans effects created in the failover window ± 5 min for duplicates by (user, amount, cart hash).

Blast radius: Every request in flight during the failover — proportional to traffic × replication lag.

Mitigation: Move the source of truth into the service DB transaction; keep Redis as a cache of completed records only. Until then, run a duplicate scan after every failover.

Prevention: A design review rule: an idempotency store that isn't transactional with the effect must declare its duplicate budget per failover and get sign-off from the effect's business owner.

Owner: The service team owns the store choice; the business owner of the effect (finance for charges) signs off on any non-transactional design.

4.3 Stuck In-Progress Records#

t=0:        Handler crashes after charging (recovery point: charged) and before final commit
t=0:        Record: status=in_progress, no lease (implementation predates locked_until)
t=+1s..+1h: Every client retry gets 409 Conflict, Retry-After: 1
t=+1h:      SDK gives up → app shows "Something went wrong"
t=+1h:      Customer was charged; no order exists. Support ticket.

Detection: idempotency.stuck_in_progress{age>5m} gauge — should be ~0. Page if > 50 for 10 min.

Blast radius: Each stuck key is one customer with an unknown outcome — money taken, nothing delivered.

Mitigation: Add locked_until + version; a retry after lease expiry takes over and resumes from recovery_point. A background resolver sweeps records stuck > 5 min with no retry, resumes them (re-calls the provider with the derived key → gets the original charge → completes the order), and pages a human queue after N failures.

Prevention: Recovery points are part of the middleware API, not optional; every handler with an external call must declare its phases.

Owner: The service team owns resolution of its stuck records; the platform owns the resolver framework and the gauge.

4.4 False Deduplication — The Key Reused for a Different Operation#

t=0:        Partner integration generates Idempotency-Key = customer_id (a "stable" value)
t=0:        First order for customer 881 executes; key 881 completed
t=+2h:      Second, different order for customer 881 → same key → fingerprint check disabled
            ("it was causing 422s") → server replays first order's 201
t=+2h:      Partner believes second order succeeded. It doesn't exist.
t=+2 weeks: Partner reconciliation: 3,100 "confirmed" orders missing. Escalation to VP.

Detection: idempotency.fingerprint_mismatch_total{client} — with fingerprinting on, this incident is 3,100 loud 422s on day one instead of silent loss. Also idempotency.replay_ratio{client} > 20% is suspicious for a partner that rarely retries.

Blast radius: Silent data loss for every client with the bug. Worse than duplicates: lost operations can't be reconciled from your side because they never reached your database.

Mitigation: Re-enable fingerprinting; contact the partner with the list of 422s. Never disable the guard because it's "noisy" — the noise is the bug report.

Prevention: The fingerprint check cannot be disabled per client without a platform exception with expiry. Partner onboarding docs show correct key generation with a code sample.

Owner: API platform owns the guard; partner engineering owns integration review.

4.5 Retry Amplification Across Layers#

t=0:        Database p99 rises from 20ms to 900ms (a slow query plan)
t=+10s:     Service B's client times out at 500ms, retries 3× → 3× load on DB
t=+20s:     Service A times out on B, retries 3× → 9×; gateway retries → 27×; SDK → 81×
t=+40s:     DB CPU 100%; every layer's retries now fail; outage
t=+40s:     Requests that DID commit before timeouts get re-executed by upper layers
            without derived keys → duplicate side effects in Service B

Detection: rpc.retry_ratio{caller,callee} > 10%; rpc.attempts_per_request p99; request fan-in at the database vs at the edge.

Blast radius: The entire call graph; the retry storm outlives the original cause.

Mitigation: Disable retries at middle layers (feature flag); shed load at the database with a circuit breaker (see Circuit Breakers).

Prevention: Retry at exactly one layer — the outermost client that holds the idempotency key. Inner layers fail fast and propagate deadlines. Retry budgets cap retries at ~10% of requests per client. Every inner call carries a derived key so the retries that do happen are safe.

Owner: The platform team owns client-library retry defaults and the retry-budget rule; service teams own their deadlines.

4.6 Consumer Dedup Window Shorter Than Replay#

t=0:        Fulfilment consumer dedupes on event_id using Redis SET with 24h TTL
t=+3 days:  Bug in consumer; operator resets consumer-group offset to 3 days ago to reprocess
t=+3 days:  Dedup keys for days 1–2 expired → 1.9M events reprocessed as new
t=+3 days:  41K orders get a second "ship" command → warehouse picks duplicates

Detection: consumer.dedup_hit_ratio drops to ~0 during a replay that should be ~100% hits; consumer.lag jump after an offset reset; alert on any manual offset reset in a consumer group tagged "side-effecting."

Blast radius: Every effect produced by the replayed range.

Mitigation: Pause the consumer; reconcile shipments against orders; resume.

Prevention: Dedup window ≥ topic retention for side-effecting consumers (inbox table in the consumer's DB, partitioned by day, dropped at retention + 1 day), or versioned/state-machine writes that are idempotent forever (UPDATE orders SET state='shipped' WHERE id=:id AND state='paid'). Offset resets on side-effecting consumers require a runbook and a second approver.

Owner: Consuming team owns its dedup; the Kafka platform owns the offset-reset guardrail.

4.7 Outbox Relay Stall or Poison Row#

t=0:        Outbox relay polls 'SELECT … WHERE published_at IS NULL ORDER BY id LIMIT 500'
t=+0s:      One event's payload exceeds broker max.message.bytes → publish fails, relay retries forever
t=+20min:   All events behind it are stuck; downstream sees no new orders; search index stale

Detection: outbox.oldest_unpublished_age_seconds > 60 → page; outbox.publish_errors_total.

Mitigation: Move the poison row to an outbox dead-letter table with an alert; relay continues.

Prevention: Payload size validated at write time; per-aggregate ordering only (not global), so one stuck aggregate doesn't block others; CDC-based relay (e.g., Debezium's outbox event router) instead of polling at high volume.

Owner: The CDC/relay platform owns stalls; the producing team owns its poison rows.

4.8 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Quoting Stripe's header ≠ designing idempotency. The record lifecycle, concurrency and propagation are the design.
• Knowing Kafka transactions ≠ exactly-once effects. Ask what the consumer does next.
• Distributed-transaction vocabulary (2PC, XA) ≠ judgment. External providers won't join your transaction; the Staff answer avoids needing them.
• "We use a dedup table" ≠ consumer idempotency — unless it's in the same transaction as the effect and outlives the replay horizon.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• The mathematical definition of idempotence beyond one sentence.
• Two-phase commit internals — say why you don't need it and move on.
• Detailed payment state machines — link Payment Processing.
• Lock-service and leader-election design — link Distributed Coordination.
• Broker internals beyond "at-least-once, redelivery on rebalance, retention-bounded replay."

6.4 Follow-Up Questions to Expect#

1. "Who generates the key, and when exactly?"
2. "Two identical requests arrive 5ms apart. What happens to each?"
3. "The first attempt crashed after charging the card. What does the retry do?"
4. "Do you replay a 500? A 402? A 429?"
5. "How long do you keep keys, and what happens to a retry after that?"
6. "How does the guarantee survive the call to the inventory service and the Kafka event?"
7. "How do you get 40 teams to do this consistently?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Make our order API safe to retry."

Drill 2: Key Semantics#

Prompt: "Who generates the key, when, and what is it unique within?"

Drill 3: Concurrent Duplicates#

Prompt: "Two copies of the same request arrive 5ms apart on different instances. Walk me through both."

Drill 4: Make It Concrete — Crash After the Side Effect#

Prompt: "The handler charges the card, then the process dies before committing the order. What happens on retry?"

Drill 5: Dependency Down — The Idempotency Store#

Prompt: "The database holding idempotency records is unavailable. Do you fail open or closed?"

Drill 6: The Hot Key — A Buggy Client Retry Loop#

Prompt: "A partner's integration retries the same key 5,000 times a second after a 409."

Drill 7: Multi-Tenant#

Prompt: "We're a platform; tenants choose their own idempotency keys. What can go wrong?"

Drill 8: Build vs Buy#

Prompt: "Should we build idempotency into the gateway, a library, or rely on the database?"

Drill 9: Policy Change Without an Outage#

Prompt: "We want to make Idempotency-Key required on all POST endpoints. 300 integrations don't send it today."

Drill 10: Multi-Region#

Prompt: "We're going active-active. A retry with the same key lands in a different region."

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Incident — Idempotency Table Becomes the Bottleneck#

Context: During a holiday sale, order-write p99 jumps from 80ms to 2.4s. The order DB's CPU is fine; lock waits on the idempotency_keys table are high. Replay rate has risen from 2% to 35% because clients are timing out and retrying. You're escalated.

Questions to Surface First:

• Are replays hitting the database, or is there a completed-record cache?
• Are retries honoring Retry-After, or is one client hammering in-progress keys?
• Is the table partitioned and its indexes healthy, or did 7-day retention at peak volume bloat it?

Typical L5 Approach: Scales the database vertically and increases client timeouts. Latency recovers partially; the retry-driven load remains and returns at the next peak.

Staff Approach: Recognizes a feedback loop: slow writes → client timeouts → retries → replay lookups and 409s on hot rows → slower writes. Breaks it: enables the completed-record cache in Redis (replays never touch the DB), confirms SDKs honor Retry-After and gateway per-key limits are on, and checks the table is partitioned by day so the hot partition's indexes fit in memory. Raises the client timeout only after the loop is broken.

Principal Approach: Treats it as a retry-policy failure across the org, not a table problem. Mandates retry budgets in all client SDKs (≤ 10% of requests), single-layer retries, and capacity planning that models retry amplification at peak (e.g., plan for 1.3× nominal write load). Adds 'replay ratio at peak' to the launch-readiness review and makes the completed-record cache part of the standard library rather than a per-team optimization.

Key Takeaway: "Retries are load. An idempotency layer that serves replays from the database turns client panic into a database incident."

What clears the Staff bar:

• Identifies the feedback loop rather than the symptom
• Breaks the loop before tuning timeouts
• Uses contract mechanisms (Retry-After, per-key limits) rather than capacity

Deep Dive 2: Silent Failure — Partners' Second Orders Vanish#

Context: A large partner reports 3,100 orders their system marked "confirmed" over two weeks that don't exist in yours. Your dashboards show no errors. Replay rate for this partner is 22%; for others, 1–3%.

Questions to Surface First:

• How does the partner generate idempotency keys?
• Is fingerprint checking enabled for this partner's traffic?
• Do the "missing" orders share keys with orders that do exist?

Typical L5 Approach: Searches logs for errors, finds none, asks the partner to resend the missing orders. Resent orders succeed. Closes the ticket. The partner's next week loses another 1,500.

Staff Approach: The 22% replay rate is the tell. Pulls replayed requests: the partner uses customer_id as the key, and fingerprint checking was disabled for them months ago after they complained about 422s. Every second order per customer was replayed as the first. Re-enables fingerprinting immediately (the partner now gets loud 422s instead of silent loss), sends them the list of affected operations with fingerprints, and fixes their key generation with partner engineering.

Principal Approach: Fixes the governance gap that allowed a safety check to be disabled per client: guard rails become non-optional without a time-boxed exception approved by the API platform owner, and every exception appears on a dashboard. Adds replay_ratio by client as an anomaly alert. Rewrites partner onboarding to include a certification test that sends two distinct operations and verifies two distinct keys.

Key Takeaway: "False deduplication is worse than duplication — the lost operation never reaches your database, so you can't reconcile it."

What clears the Staff bar:

• Uses replay ratio as a diagnostic signal
• Recognizes the disabled guard as the root cause, not the partner bug alone
• Converts silent loss into loud errors immediately

Deep Dive 3: Large-Customer Onboarding — An Enterprise With a 7-Day Retry Horizon#

Context: An enterprise customer's ERP integration batches 2M mutating API calls nightly and, on any failure, re-runs the entire failed batch for up to 7 days. Your idempotency TTL is 24h. They're about to go live.

Questions to Surface First:

• How does the ERP generate keys — per record, per batch, per attempt?
• Does each record have a natural business key (their document number)?
• What volume of storage does 7-day retention at 2M/night imply?

Typical L5 Approach: Raises the global TTL to 7 days for everyone. Storage grows 7× for all tenants to serve one.

Staff Approach: Uses the natural key: the ERP's document number, unique per tenant, becomes the idempotency key via a unique constraint on (tenant, external_ref) — dedup is permanent, not TTL-bound, and costs one index. For endpoints without a natural key, offers per-tenant retention (7d for this tenant: 2M × 2 KB × 7 ≈ 28 GB) and a bulk API that accepts a batch ID plus per-item keys, returning per-item outcomes so partial failures re-run only the failed items.

Principal Approach: Turns retention into a product decision: retention tiers (24h standard, 7d enterprise) priced into plans, with the storage cost visible. Makes 'natural external reference as idempotency key' the recommended integration pattern in the enterprise integration guide, because it is stronger and cheaper than any TTL. Adds bulk-API per-item idempotency to the platform roadmap since every ERP customer will need it.

Key Takeaway: "The strongest idempotency key is the customer's own business identifier — it never expires."

What clears the Staff bar:

• Prefers natural keys over TTL extension
• Sizes storage per tenant
• Designs partial-failure semantics for bulk operations

Deep Dive 4: Post-Mortem — 41,000 Double Shipments After an Offset Reset#

Context: A fulfilment consumer had a bug; an operator reset its Kafka consumer-group offset by 3 days to reprocess. The consumer deduped on event_id in Redis with a 24h TTL. 41,000 orders shipped twice. Cost: ~$1.2M in goods and logistics.

Questions to Surface First:

• Why was the dedup window shorter than the topic's retention?
• Why was a manual offset reset on a side-effecting consumer possible without review?
• Is the "ship" effect itself idempotent at the warehouse (does it accept a shipment ID)?

Typical L5 Approach: Extends the Redis TTL to 7 days. Next replay beyond 7 days, or the next Redis eviction under memory pressure, repeats the incident.

Staff Approach: Makes the effect idempotent forever rather than for a window: the consumer transitions orders with UPDATE orders SET state='shipping', shipment_id=:sid WHERE id=:id AND state='paid' and calls the warehouse with shipment_id derived from order_id, which the warehouse dedupes. The inbox table moves into the consumer's DB, same transaction, retained ≥ topic retention. Offset resets on side-effecting consumers require a runbook and second approver.

Principal Approach: Classifies consumers org-wide as side-effecting or pure, and requires every side-effecting consumer to pass a 'full replay' test in CI: replay the entire retention window and assert no new external effects. Puts the offset-reset guardrail in the Kafka platform, not each team's runbook. Takes the $1.2M to leadership as the price of 'dedup with a TTL' and funds the paved-road inbox library.

Key Takeaway: "A dedup window is a bet on how far back anyone will ever replay. State-guarded effects don't need to win that bet."

What clears the Staff bar:

• Replaces windowed dedup with naturally idempotent state transitions
• Pushes idempotency to the external effect (warehouse shipment ID)
• Adds an operational guardrail on the human action that triggered it

Deep Dive 5: Multi-Region Expansion#

Context: The company is moving the order API to active-active in two regions. Today the idempotency table lives in the single-region order database.

Questions to Surface First:

• Is the data model entity-homed (each tenant/order has a home region) or truly multi-writer?
• How does the client route — anycast, geo-DNS, or tenant-aware?
• During a regional failover, what happens to records in in_progress?

Typical L5 Approach: Replicates the idempotency table with an async multi-region table and assumes conditional writes still work. They do — per region. A retry that lands in the other region within the replication lag executes again; last-writer-wins later overwrites one of the two records.

Staff Approach: Keeps writes single-homed per tenant: the edge routes mutating requests to the tenant's home region; the idempotency record lives with the tenant's data. On failover, the home moves with the data; in_progress records are taken over in the new region via the normal lease-expiry path, and their recovery points tell the resolver which downstream calls to re-issue with derived keys.

Principal Approach: Makes idempotency a first-class requirement of the org's multi-region architecture review: any multi-writer data model must state how idempotency records are homed. Publishes 'tenant-homed writes' as the default cell model, with globally consistent stores reserved for the few entities that need them, priced (~100ms per write) and approved explicitly.

Key Takeaway: "Asynchronously replicated dedup is regional dedup. Home the key with the data."

What clears the Staff bar:

• Spots the last-writer-wins trap in async multi-region tables
• Aligns idempotency with the data-homing model
• Defines failover behavior for in-flight operations

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Name the four sources of duplicates and the mechanism that handles each
• Explain why exactly-once delivery is impossible and how exactly-once effect is assembled
• Specify an idempotency key contract: who mints it, when, scope, fingerprint, TTL, expiry behavior, status codes
• Design the idempotency record's lifecycle including in-progress leases, takeover, fencing and recovery points
• State which outcomes to replay and which to release, and why the 500-after-side-effect case is neither
• Carry idempotency across a downstream call (derived keys), a broker (outbox, inbox) and a replay (window ≥ retention or state-guarded writes)
• Explain retry amplification and the single-layer retry rule with retry budgets
• Make idempotency a platform default: middleware, SDKs, gateway enforcement, durability tiers

The Bar for This Question#

Mid-level (L4): Knows that POST isn't idempotent and that retries can duplicate. Proposes "check if it already exists." Doesn't reach concurrency or crash windows.

Senior (L5): Proposes Idempotency-Key with a Redis store and TTL, response caching, and exponential backoff. Correct for the simple retry. Under probing, concurrent duplicates, the crash-after-effect window, Redis failover and downstream propagation are improvised.

Staff+ (L6): Frames the problem as unknown outcomes and exactly-once effect. Defines the key contract precisely, co-locates the record with the effect, handles concurrent and crashed attempts with leases, fencing and recovery points, and carries the guarantee through derived keys, outbox and consumer inbox with a window that outlives replay. Has metrics for every leak and an owner for every one. The interviewer should learn something from the answer — often the false-deduplication failure or the expiry-rejection trick with timestamped keys.

10. Staff Insiders: Controversial Opinions#

10.1 "Exactly-Once Delivery Is a Marketing Term"#

The Staff position: Say "exactly-once effect" and name the dedup mechanism at each hop. Never say "exactly-once delivery" without a qualifier.

Why this matters in interviews: It's the fastest way to show you know where guarantees end.

10.2 "Redis Is the Wrong Default Idempotency Store"#

The Staff position: Source of truth in the service DB, in the effect's transaction. Redis in front, for completed replays only. Redis-only is acceptable for an explicitly low-stakes tier.

Why this matters in interviews: The Redis answer is the most common L5 answer. Explaining precisely why it leaks — and where Redis still belongs — is a clean level signal.

10.3 "A Fixed TTL Is a Bug Waiting for a Slow Client"#

The Staff position: TTL is half of a contract whose other half is the client's retry horizon. Publish both, enforce both, and make expiry an error, not a re-execution.

Why this matters in interviews: "24 hours, like Stripe" is cargo cult; the follow-up "and a client that retries at 30 hours?" separates levels.

10.4 "Most Idempotency Bugs Are Retry-Policy Bugs"#

The Staff position: Before designing deduplication, find out how many layers retry. Remove all but one.

Why this matters in interviews: Connecting idempotency to retry policy across the call graph shows system-wide thinking.

10.5 "False Deduplication Is Worse Than Duplication"#

The Staff position: Fingerprinting is not optional. A 422 for key reuse is the system working.

Why this matters in interviews: Most candidates only design against duplicates. Naming the opposite failure is rare and memorable.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer makes one API safe to retry. A Principal engineer notices that idempotency is a property of the weakest hop: the order API can be perfect, and the customer is still double-charged because the payments client library retries without a derived key, or the fulfilment consumer dedupes with a 24h TTL, or a service mesh adds a retry nobody configured. Across 40 teams there will be 6 storage choices, 5 TTLs, 4 sets of status codes and 3 SDKs that handle 409 differently. At L7 the problem is the org's retry-and-dedup contract: one key standard, one middleware library, one SDK behavior, one retry policy, one outbox/inbox paved road, and a metric — unknown outcomes and duplicate effects — that leadership can see.

🧭 Principal Move: "Before we design this endpoint, how many layers in our stack retry by default, and how many idempotency implementations do we have? I'd expect the answers to be 'four' and 'six', and the highest-leverage work is making it one of each."

The Org-Level Fault Line#

Central idempotency service vs per-team implementations vs platform library + contract.

The deciding question: can the record commit with the effect? If yes — and for every service with its own database it can — the platform ships code, not a service.

Cost Model#

Assumptions: loaded engineer cost ~$25K/month; Postgres storage ~$0.10–0.25/GB-month with replicas; Redis cache ~$200–$400/node-month; average record 2 KB with response; duplicate-effect cost estimated from refunds and support time.

The line that matters to leadership: The infrastructure is cheap; the absence of it is expensive. A single incident like Deep Dive 4 ($1.2M) or a month of 0.1% duplicate charges on $50M monthly volume ($50K in refunds plus support and chargeback fees) pays for the whole platform team for a year.

The 3-Year Evolution Path#

Each step is triggered by evidence. Building the inbox paved road before any team has a side-effecting consumer is premature; building it after the second replay incident is late.

One-Way Doors vs Two-Way Doors#

🧭 Principal Insight: The status codes, header name, scope and TTL are a public API. They get a design review with API governance and partner engineering. The storage engine gets a code review.

The Standard I'd Write#

RFC: Idempotency and Retry Standard (v1)

Scope: Every externally reachable mutating endpoint and every internal endpoint that causes a side effect outside its own database.

Requirements:

• Mutating endpoints MUST accept Idempotency-Key (16–64 chars), scoped to (tenant, endpoint), and MUST reject reuse with a different request fingerprint (422), concurrent use (409 + Retry-After), and keys older than the published retention (idempotency_key_expired).
• Idempotency records MUST commit in the same transaction as the local effect. Separate stores are permitted only for endpoints in the best-effort tier.
• Every call to another service or vendor that causes a side effect MUST carry a derived key and be bracketed by recovery points.
• Retries MUST occur at exactly one layer (the outermost key holder). Inner layers MUST fail fast and propagate deadlines. Client SDKs MUST enforce a retry budget of ≤ 10%.
• Events MUST be published via outbox or CDC; dual writes are blocked by lint.
• Side-effecting consumers MUST dedupe in the same transaction as their effect with a window ≥ topic retention, or use state-guarded writes, and MUST pass a full-replay test.
• Services MUST emit idempotency.replay_total, idempotency.fingerprint_mismatch_total, idempotency.stuck_in_progress, and ops.unknown_outcome_total.

Exceptions: Filed with the API platform group; decision within 5 business days; expiry ≤ 2 quarters; listed on a public exceptions dashboard.

Success metrics: 100% of money-moving endpoints compliant in 2 quarters, all mutating endpoints in 4; duplicate-effect incidents → 0 per quarter; unknown outcomes older than 24h < 0.001% of operations; rpc.retry_ratio < 10% fleet-wide.

What I'd Tell the VP#

"Every time a network hiccups, our systems have to guess whether an action already happened, and right now six teams guess six different ways. That's why we had the double-shipment incident last quarter and why some partners see 'confirmed' orders that don't exist. I'm proposing one standard for how every API handles retries and one shared library that implements it, rolled out to money-moving endpoints first. It costs two to three engineers for a year. It removes the incident class that cost us $1.2M in one night, and it gives us a single number — operations with unknown outcomes — that tells us whether we're getting it right."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll store keys in the same transaction as the effect." — correct, but only for this service; silent on the 39 others and how they adopt it.
• "Consumers dedupe with an inbox table." — correct, but no replay test, no offset-reset guardrail, nothing that stops the next team from using a 24h Redis TTL.
• "Retry with the same key." — correct, but never asks how many layers retry or who sets the retry budget.

Appendices

CREATE TABLE idempotency_keys (
  tenant_id        BIGINT       NOT NULL,
  endpoint         TEXT         NOT NULL,          -- e.g. 'POST /v1/orders'
  idem_key         TEXT         NOT NULL,          -- 16..64 chars
  fingerprint      BYTEA        NOT NULL,          -- SHA-256 of canonical request
  status           TEXT         NOT NULL,          -- 'in_progress' | 'completed'
  recovery_point   TEXT         NOT NULL DEFAULT 'started',
  version          INT          NOT NULL DEFAULT 1, -- fencing for takeover
  locked_until     TIMESTAMPTZ  NOT NULL,
  response_code    INT,
  response_body    BYTEA,                         -- capped at 16 KB
  resource_ref     TEXT,                          -- e.g. 'ord_9' for debugging
  created_at       TIMESTAMPTZ  NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, endpoint, idem_key)
) PARTITION BY RANGE (created_at);               -- daily partitions, drop at TTL + 1 day

function handle(req):
  key = req.header("Idempotency-Key")       or return 400 missing_key
  if key_timestamp(key) < now() - TTL:      return 422 idempotency_key_expired
  fp  = sha256(canonical(req.method, req.path, req.body_without_volatile_fields))
  scope = (req.auth.tenant_id, req.route, key)

  row = INSERT … status='in_progress', locked_until=now()+LEASE
        ON CONFLICT DO NOTHING RETURNING *
  if row is null:                                        # someone has it
      existing = SELECT … WHERE scope
      if existing.fingerprint != fp:     return 422 idempotency_key_reused
      if existing.status == 'completed': return replay(existing)   # + Idempotent-Replayed: true
      if existing.locked_until > now():  wait ≤ 1s for completion, else return 409 Retry-After: 1
      row = UPDATE … SET locked_until=now()+LEASE, version=version+1
            WHERE scope AND locked_until < now() AND version = existing.version RETURNING *
      if row is null:                    return 409 Retry-After: 1   # lost takeover race

  return run_phases(row, req)

function run_phases(row, req):
  try:
    if row.recovery_point == 'started':
        tx: insert order(pending_payment); set recovery_point='order_created' WHERE version=row.version
    if row.recovery_point == 'order_created':
        result = psp.charge(key = row.idem_key + ":charge", …)   # downstream dedupes
        if result.declined:
            tx: order→declined; complete(row, 402, body)            # deterministic outcome, final
            return 402
        tx: order→paid; outbox.insert(OrderPlaced); complete(row, 201, body) WHERE version=row.version
    return replay(row)
  except BeforeAnySideEffect as e:
    DELETE row WHERE scope AND version=row.version                  # release: nothing happened
    raise e
  except Unknown as e:                                              # timeout, crash, 5xx downstream
    leave row in_progress at its recovery point
    return 503 Retry-After: 2                                       # client retries same key

root      = client key k                          (e.g. 0190f3a2-…)
inventory = k + ":reserve"
payment   = k + ":charge"
refund N  = k + ":refund:" + N                    (N from our own state, not attempt count)
email     = k + ":email:order_confirmation"

op = Operation(key = uuidv7(), request, created_at = now())
persist(op)
for attempt in 1..MAX_ATTEMPTS (3):
    if now() - op.created_at > TTL - 1h: surface UNKNOWN to caller; stop
    if !retry_budget.allow(): surface UNKNOWN; stop
    resp = send(op.request, Idempotency-Key = op.key, deadline = remaining_budget)
    match resp:
      2xx, 4xx except 409/429 → terminal; delete(op); return resp
      409, 429, 503           → sleep(max(Retry-After, backoff(attempt) with full jitter))
      network timeout         → sleep(backoff(attempt) with full jitter)
surface UNKNOWN with op.key   # caller can poll GET /operations/{key} or retry later

# Contract health
idempotency.requests_with_key_ratio{endpoint}         # coverage
idempotency.replay_total / replay_ratio{client}
idempotency.conflict_409_total{endpoint}
idempotency.fingerprint_mismatch_total{client}
idempotency.expired_key_rejections_total
idempotency.key_age_at_request_seconds (histogram)

# Correctness health
idempotency.stuck_in_progress{age_bucket}
idempotency.takeover_total
ops.unknown_outcome_total{age_bucket}
dup_detector.suspected_duplicates_total{effect}       # same user + amount + cart within 10 min, different keys

# Pipeline health
outbox.oldest_unpublished_age_seconds
consumer.dedup_hit_ratio{consumer}
rpc.retry_ratio{caller,callee}

records/day × avg_bytes × retention_days
10M × 2 KB × 3   ≈ 60 GB       (72h)
10M × 2 KB × 7   ≈ 140 GB      (7d)
500M × 2 KB × 3  ≈ 3 TB        (72h, org-wide)