Technologies referenced in this case study: PostgreSQL · Apache Kafka · Redis · DynamoDB · API Gateways

Related: Reservation Systems · Flash Sales & Ticketing · Dealing with Contention · Managing Long-Running Processes · Consistency Models · API Design Patterns

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once, then return to the fault lines and deep dives that match your weak spots.

Without idempotency + ledger:
t=0:       User clicks "Pay $120". Checkout service calls PSP.
t=+8s:     PSP authorizes the card. Response is lost (LB reset).
t=+10s:    Client timeout. Service returns 500. User sees "Payment failed".
t=+15s:    User clicks "Pay" again. New PSP call. Second $120 authorization + capture.
t=+2 days: User sees two charges on the statement. Files a dispute.
t=+30 days: Chargeback: $120 refunded + $15 dispute fee. Support ticket. Trust gone.
            Finance finds a $120 hole nobody can explain for 3 weeks.

With idempotency keys + state machine + reconciliation:
t=0:       Checkout creates payment_intent pi_123 (state=CREATED) with key k_abc.
t=+8s:     PSP authorizes. Response lost.
t=+10s:    Timeout → pi_123 moves to UNKNOWN, not FAILED. User sees "Confirming…".
t=+15s:    Retry with the SAME idempotency key k_abc → PSP returns the original result.
t=+15s:    pi_123 → AUTHORIZED. Ledger posts balanced entries. User sees "Paid".
t=+1 day:  Nightly reconciliation matches PSP settlement file to ledger: 0 breaks.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What This Interview Actually Tests#

A payment system is not an API integration question. Everyone can call Stripe.

It is a correctness-under-uncertainty question that tests:

• Whether you treat "timeout" as a third outcome, distinct from success and failure
• Whether you can make retries safe without distributed transactions
• Whether you know the difference between your record of money and the PSP's record — and who wins when they disagree
• Whether you design for the auditor, the finance team and the on-call — not only the checkout button

The key insight: You cannot get exactly-once money movement across a network. You get at-least-once delivery + idempotent effect + reconciliation to catch the residue. Staff candidates name all three; Senior candidates name the first one and call it done.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

Three intents produce three different systems. Name them, then commit.

🎯 Staff Move: "I'll assume we're a merchant integrating PSPs for checkout — pay-in only — and I'll design the ledger so payouts can be added later. If we were a marketplace, the ledger becomes the center of gravity; if we were the processor, this is a different interview about ISO 8583 and HSMs. Tell me if you want one of those instead."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Naming how real payment companies solved these problems shows you've studied operations, not tutorials.

Stripe — Idempotency Keys as a Public API Contract#

Stripe's API accepts an Idempotency-Key header on every POST. The server stores the key with the first request's result and returns the same response for replays — including errors — and rejects a replay whose parameters differ from the original. Keys are retained for at least 24 hours. Stripe's engineering writing on idempotency describes the server side: record the key and request in a transaction, split the work into atomic phases with recovery points, and have a background process complete or roll back abandoned requests.

Staff insight: Idempotency is not "dedup on the server". It is a contract between client and server: the client promises to reuse the key for the same intent, the server promises the same answer. In an interview, say both halves.

Airbnb — Orpheus, Idempotency as a Library#

Airbnb publicly described avoiding double payments in their SOA migration with an idempotency library (Orpheus). Each request is split into three phases — pre-RPC (record intent in the DB), RPC (call the external processor), post-RPC (record the result) — and the library forbids mixing database writes into the network phase. Failures are classified as retryable or non-retryable, and the classification is persisted so a retry never re-does a non-retryable failure.

Staff insight: The design point is separating local transactions from network calls. You cannot hold a DB transaction open across a 3-second PSP call. Name the three phases and you've shown you've built this.

Uber — Ledger-First Money Movement#

Uber has written publicly about its payments platform and its purpose-built ledger storage (LedgerStore), designed for immutable, verifiable ledger entries at very high volume, including sealing of time ranges so historical entries cannot change. Money movement at Uber is modeled as ledger entries between accounts (riders, drivers, Uber) rather than as mutable balances.

Staff insight: At scale the ledger becomes its own storage system with its own guarantees — immutability, completeness checks, time-range sealing. In an interview, you don't build LedgerStore; you say "the ledger's invariants are append-only and sum-to-zero, and I'd pick storage that makes violating them hard."

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: The hot path is short on purpose: gateway → payment service → risk → PSP. Everything that can be async is: ledger posting (via outbox + Kafka), webhook ingestion, resolution of UNKNOWN intents, and reconciliation. The two numbers that matter most on the observability panel are payment.unknown_count (how much money is in limbo right now) and ledger.imbalance_total (must be exactly zero; any non-zero value is a Sev-1).

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 20 minutes on the checkout API and the PSP call, then run out of time before the interviewer asks the only question that matters: "The call timed out. Did we charge them?" Compress the basics to ~10 minutes and spend the rest on unknown outcomes, the ledger and reconciliation.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional scope in one breath:

"Users pay for orders with cards and wallets. We authorize at checkout, capture on shipment, support full and partial refunds, and handle disputes. We integrate PSPs — we are not the processor."

Then spend the time on non-functional requirements, which is where the design lives:

"Three constraints drive everything. One: no double charges and no lost charges — correctness beats latency. Two: every cent must be explainable to finance, so I need an auditable record, not a status column. Three: we depend on an external PSP whose p99 is seconds and whose outages we don't control. I'll also assume roughly 200 payments per second at peak, which means throughput is not the hard part."

Then name the underspecified parts:

"A few things I'd confirm: single currency or multi-currency? Are we a marketplace paying out to sellers? Do we store cards for repeat purchase? I'll assume single currency to start, merchant-of-record, stored cards via PSP tokens."

🎯 Staff Move: Saying "throughput is not the hard part" early is a strong signal. It tells the interviewer you won't waste 10 minutes sharding a table that handles 200 TPS, and it redirects the session to correctness.

Phase 2: Core Entities & API (1–2 minutes)#

Name the nouns in 30 seconds:

• PaymentIntent: intent_id, order_id, amount_minor, currency, state, psp, idempotency_key, version
• PaymentAttempt: one row per PSP call — attempt_id, intent_id, operation (authorize/capture/refund/void), psp_reference, outcome (SUCCEEDED / DECLINED / UNKNOWN), request_hash
• LedgerEntry: txn_id, account_id, direction (debit/credit), amount_minor, currency, caused_by_event_id, posted_at
• IdempotencyRecord: key, scope, request_hash, response_blob, status, expires_at

Money is always stored as integer minor units (cents) with an explicit currency. Never floats.

Client-facing API:

POST /v1/payment_intents                Idempotency-Key: <uuid>
     { order_id, amount_minor, currency, payment_method_token }
  → 201 { intent_id, state: "REQUIRES_CONFIRMATION" }

POST /v1/payment_intents/{id}/confirm   Idempotency-Key: <uuid>
  → 200 { state: "AUTHORIZED" | "DECLINED" | "PROCESSING" }

POST /v1/payment_intents/{id}/capture   Idempotency-Key: <uuid>   { amount_minor? }
POST /v1/payment_intents/{id}/refunds   Idempotency-Key: <uuid>   { amount_minor, reason }
GET  /v1/payment_intents/{id}

PROCESSING is the honest client-facing name for UNKNOWN. The client polls GET or receives a push; it never re-submits a new intent.

🎯 Staff Move: "I'm separating intent creation from confirmation. The intent exists before any money moves, so the idempotency key and the order linkage are durable before the first network call to the PSP. That ordering is the whole trick."

Phase 3: High-Level Architecture (≤5 minutes)#

Draw at most eight boxes:

Walk the confirm flow in 90 seconds:

1. Client confirms intent pi_123 with idempotency key k1. Gateway requires the header.
2. Payment service looks up k1. Hit with same request hash → return stored response. Hit with different hash → 422. Miss → insert record as IN_PROGRESS (row lock).
3. Risk scores the payment (p99 < 100ms). Decline → terminal.
4. Pre-call: write PaymentAttempt(operation=authorize, outcome=PENDING) and commit.
5. Call: PSP authorize with PSP-level idempotency key pi_123:authorize:1.
6. Post-call: in one transaction — update attempt outcome, move intent state, write outbox event, store idempotency response.
7. Outbox relay publishes payment.authorized; ledger consumer posts balanced entries.
8. On timeout at step 5: intent → UNKNOWN, respond PROCESSING, resolver owns it from here.

🎯 Staff Move: Say out loud: "Notice there's no database transaction open across the PSP call. Pre-call commit, network call, post-call commit. If we crash between them, the PENDING attempt row tells the resolver exactly what to go check." You've now spent ~9 minutes.

Phase 4: Transition to Depth (1 minute)#

"That's the happy path, and it's the Senior-level design. What makes this hard is that the PSP call can fail after money moved. I'd like to go deep on three things: how we resolve unknown outcomes without double-charging, how the ledger stays balanced and auditable, and how reconciliation catches what everything else misses. Where would you like to start?"

If no preference: start with the unknown outcome. It's the question that decides the level.

Phase 5: Deep Dives (25–30 minutes)#

For each: state the tradeoff → commit → quantify → name who pays.

Deep dive 1: The unknown outcome (7–8 min)

"A timeout tells us nothing. The PSP may have authorized, declined, or never received the request. Three rules. First, the intent goes to UNKNOWN and the user sees 'Confirming your payment', not 'failed'. Second, the resolver retries the identical call with the identical PSP idempotency key at 1 minute, 5 minutes and 30 minutes — the PSP either returns the original result or processes it for the first time, and both are correct. Third, if the PSP supports lookup by our reference, we query instead of retrying. After 24 hours unresolved, it goes to a human queue owned by payments ops."

Quantify: "Normally ~0.01% of calls end UNKNOWN — at 200 TPS that's ~2 per 1,000 seconds, trivially handled. During a PSP brownout it can hit 5%, which is 10/sec, 36,000/hour. The resolver must be rate-limited so it doesn't DDoS a recovering PSP, and the UX must hold 'Confirming' for minutes."

Who pays: "The customer pays a few minutes of uncertainty. That's the right victim — the alternative victim is the customer charged twice."

Deep dive 2: The ledger (7–8 min)

"State machine answers 'what stage is this payment in'. The ledger answers 'where is the money'. They're different questions. For an authorized-then-captured $100 order with a 2.9% + 30¢ fee:"

txn_1 (capture):   DEBIT  psp_receivable        10000
                   CREDIT customer_payments     10000   -- revenue-pending
txn_2 (fee):       DEBIT  psp_fees                320
                   CREDIT psp_receivable          320
txn_3 (settle):    DEBIT  bank_cash              9680
                   CREDIT psp_receivable         9680
                   -- psp_receivable now nets to 0: settled in full

"Every transaction sums to zero. psp_receivable should net to zero once settlement lands — if it doesn't after T+3, that's a reconciliation break with a name. Corrections are new reversing transactions, never UPDATEs."

Deep dive 3: Reconciliation (5–6 min)

"Three-way: our ledger vs the PSP's settlement report vs the bank statement. Match on PSP reference and amount. Unmatched on either side is a break. Breaks have a type — missing in ledger, missing at PSP, amount mismatch, currency mismatch — and each type has an owner and an auto-resolution rule. Target: < 0.01% of transactions break; breaks older than 2 business days page the finance-ops lead."

Deep dive 4: Events without dual writes (3–4 min)

"The payment service writes the state change and an outbox row in one Postgres transaction. A relay tails the outbox and publishes to Kafka with the outbox row ID as the event ID. Consumers — ledger, notifications, fulfillment — dedup by event ID. Crash anywhere, and the worst case is a duplicate publish, which consumers absorb."

Deep dive 5: Ownership (3–4 min)

"Payments team owns the state machine, PSP integrations and the resolver, and carries the pager for payment.unknown_count. Finance-ops owns reconciliation breaks. Risk owns fraud rules and the fail-open/closed decision for the risk engine — with a CFO-level sign-off because it's a direct loss decision. Product teams never call a PSP directly."

Phase 6: Wrap-Up (2–3 minutes)#

"The core idea: exactly-once money movement isn't available, so we build exactly-once effect from three parts — idempotency keys that make retries safe, a state machine that treats timeouts as unknown, and reconciliation that catches the residue. The ledger makes the whole thing auditable."

The evolution closer:

"What I'd build later: a second PSP when GMV justifies it, which I'd estimate around $20–50M/day; multi-currency with FX as ledger entries; payouts if we become a marketplace. What I'd never build: our own card-network connectivity, unless payments becomes the product."

🎯 Staff Move: End on who owns the residue. Every Senior candidate ends with "and we'd add monitoring." Staff candidates end with "and finance-ops owns reconciliation breaks with a 2-day SLA, because that's where the money that slips through our mechanisms gets found."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Payments is the interview where "it's eventually consistent" is not an acceptable answer by itself and "use a distributed transaction" is not available. The PSP will not join your two-phase commit. The candidate must invent correctness from local transactions, idempotency and after-the-fact verification — and must name who owns each leak. That is Staff work: reasoning about systems you don't control and outcomes you can't observe directly.

It also has an unusually expensive failure distribution. A lost cache entry costs milliseconds. A duplicate charge costs a chargeback ($15–25 fee plus the amount), a support ticket ($5–15 to handle), and — at the network level — dispute ratios that can trigger monitoring programs. A missing charge is revenue given away. Both are silent: no error fires when a customer is charged twice.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"The PSP call timed out after 10 seconds. The user is looking at a spinner. What do you show them, what state is the payment in, and what happens in the next 24 hours?"

A candidate who answers with a state name (UNKNOWN), a user-facing message ("Confirming"), a resolution mechanism (same-key retry or lookup), a time bound (24h to human queue) and an owner (payments ops) has operated a payment system. A candidate who says "retry with backoff" has integrated one.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Merchant checkout (pay-in) → conversion + no duplicates

• Constraint: checkout latency adds directly to abandonment; ~1s of added latency is a measurable conversion hit
• Strategy: PSP-hosted tokenization, two-step auth/capture, idempotent intents, internal ledger for revenue, refunds and fees
• Failure mode: double charge on retry, lost sale on a timeout shown as failure
• Who pays for imperfection: customers (duplicates) or the business (lost sales, chargeback fees)

Marketplace (pay-in + payout) → the ledger is the product

• Constraint: you hold other people's money; payouts, KYC, 1099/tax reporting, possibly money-transmitter licensing depending on structure
• Strategy: ledger with accounts per buyer, seller and platform; funds flow buyer → holding → seller minus fees; payouts scheduled after risk holds (e.g., 24h after check-in for lodging)
• Failure mode: paying out before collecting, negative seller balances after chargebacks
• Who pays: the platform eats every negative balance it can't recover

Being the processor → network protocol and regulatory engineering

• Constraint: ISO 8583 messages to card networks, HSMs for PIN/key management, PCI DSS Level 1 (>6M transactions/year), four-nines availability
• Strategy: an entirely different system — acquiring, authorization switching, clearing files, settlement
• Failure mode: network-wide authorization failures, settlement file mismatches in the millions
• Who pays: every merchant on the platform

2.2 When NOT to Build a Payment System#

• You have one product, one currency, one PSP and < $1M/month. Use the PSP's hosted checkout (Stripe Checkout, Adyen Drop-in) plus their dashboard. Your "payment system" is a webhook handler and an orders.paid_at column. Building an internal ledger here is ~2 engineer-quarters for no auditable benefit.
• You want a ledger for non-money things (loyalty points you can't redeem for cash, internal credits). A double-entry ledger is still a good model, but you don't need PSP reconciliation, PCI scope or UNKNOWN resolution — don't import the machinery.
• Subscriptions. Billing (plans, proration, dunning, invoices) is a separate system on top of payments. Buy it (Stripe Billing, Chargebee, Zuora) until billing logic is a competitive advantage.
• You're tempted to become a payment facilitator to save 0.3% in fees. Payfac status brings underwriting, KYC, risk and liability for sub-merchants' chargebacks. That's a company decision, not an architecture decision.

🎯 Staff Insight: "The cheapest payment system is the PSP's hosted page. I'd build our own intent/ledger layer only when we need something the PSP can't give us: multi-PSP routing, marketplace splits, or finance-grade books across processors."

2.3 What the Interviewer Leaves Underspecified#

Interviewers deliberately omit:

• Who holds the money (merchant vs marketplace vs processor) — changes the ledger and the licensing
• When the user is told "paid" — at authorization, capture or settlement
• Currency and geography — multi-currency means FX entries; the EU means Strong Customer Authentication (3-D Secure) and asynchronous confirmation flows
• Stored credentials — whether you keep cards on file, which changes PCI scope and adds network tokens
• Refund and dispute policy — partial refunds and chargebacks are where mutable-status designs break
• Peak shape — a flash sale is 50× baseline for 10 minutes; a subscription renewal batch is 100× at 00:00 UTC

Staff engineers surface these and commit. Senior engineers assume them away and get surprised by the follow-up.

2.4 Precise Terminology#

🎯 Staff Insight: If the interviewer says "when the payment succeeds", ask: "Succeeds at which stage — authorization, capture, or settlement? They happen days apart and fail differently."

3. The Five Fault Lines#

Every payment design decision has a technical side (what state, which store, which retry) and an organizational side (who signs off on losses, who works the break queue, who gets paged). Interviewers grade the second side.

3.1 Fault Line 1: Retry vs Reconcile — The Unknown Outcome#

The tension: Resolving an UNKNOWN quickly (aggressive same-key retries, synchronous lookup) gets the customer an answer in seconds but loads a PSP that may be failing. Waiting for webhooks or reconciliation is safe but can leave a customer staring at "Confirming" for hours — or leave an order unshipped.

Staff default: "Layered. One inline retry with the same key after 2 seconds if the checkout budget allows; otherwise hand to the async resolver at 1m, 5m, 30m, 2h; webhooks resolve whichever comes first; reconciliation is the backstop. After 24 hours unresolved, a human queue."

When to deviate:

• In-person/POS payments: the customer is at the till — resolve synchronously or void-and-retry with explicit reversal messages, because "check back later" isn't a UX.
• Low-value, high-volume (micro-transactions): skip inline retry entirely; rely on async + batch reconciliation. The cost of a stuck $0.99 purchase is below the cost of the extra PSP load.
• PSP without idempotency support: lookup-by-reference before any retry. If neither is supported, you cannot retry safely at all — escalate that as a vendor-selection problem.

🧭 Principal Move: "I'd make 'unknown' a first-class concept across every money-moving system in the company — payments, payouts, refunds, gift cards. Same state name, same resolver framework, same dashboard of money-in-limbo by system. The metric money_in_unknown_usd becomes a finance KPI, not a payments-team internal."

❌ Common L5 Trap: "If the call times out, we'll retry with exponential backoff up to 3 times." The interviewer asks: "With a new request each time?" If the key is generated per HTTP request — a common bug in hand-rolled clients — each retry is a new charge. The fix is structural: the key is derived from the persisted attempt row, so a retry cannot produce a new key.

3.2 Fault Line 2: Ledger vs PSP as Source of Truth#

The tension: The PSP knows what the card network did. You know why (which order, which refund, which fee split). Neither alone can answer "what does the business have?" When they disagree, someone has to decide which record is corrected.

Staff default: "Internal ledger is the truth for our books; the PSP is the truth for what happened on the network. When they disagree, we never edit either — we post an adjusting entry in our ledger with a reference to the break ticket. The break type decides the owner."

When to deviate:

• Early stage, one PSP, no marketplace: PSP-as-truth with nightly export is fine. Write the intent/state machine now, defer the ledger until month-end close takes more than 3 days or you add a second PSP.
• Regulated entities (e-money, banking): the ledger must be the legal record with safeguarding reconciliation daily — deviation is in the direction of more rigor, sometimes intraday.

🧭 Principal Move: "Month-end close time is the metric I'd use to justify the ledger to the CFO. If finance spends 8 days closing because payments data doesn't reconcile, a ledger that gets it to 2 days pays for its team."

❌ Common L5 Trap: "We'll compute the merchant balance with SELECT SUM(amount) FROM payments WHERE status='succeeded'." Then: a $100 payment is partially refunded $30, disputed, the dispute is won, and there's a 2.9% fee. What's the balance? The mutable-status model has no answer that survives an audit. The ledger answer is "sum the entries in the account", and every entry has a cause.

3.3 Fault Line 3: Synchronous Confirmation vs Asynchronous Settlement#

The tension: Users and product want "Paid ✓" instantly. Money isn't final until capture (hours–days) or settlement (T+1–T+3), and 3-D Secure, bank transfers and wallets confirm asynchronously anyway.

Staff default: "The order is 'placed' at authorization; money is 'collected' at capture; the ledger recognizes revenue per finance's policy. The UI never says 'paid' for an UNKNOWN. Every transition is a conditional update — UPDATE … SET state='CAPTURED', version=version+1 WHERE id=? AND state='AUTHORIZED' AND version=? — so a duplicate webhook can't move a state twice."

When to deviate:

• Digital goods delivered instantly: use sale (auth+capture) and treat authorization as "paid". Refund risk is lower than the UX cost of a pending state.
• Long fulfillment (> 7 days, e.g., pre-orders): hold expiry forces either re-authorization (which can fail) or charge-at-order with refund on cancellation. Pick per product with finance.

🎯 Staff Insight: "The state machine is the contract between payments and every other team. Fulfillment listens for CAPTURED, not AUTHORIZED. Support tooling shows UNKNOWN distinctly. If I let each team infer state from different events, I've created five state machines."

3.4 Fault Line 4: Single PSP vs Multi-PSP Routing#

The tension: One PSP means one integration, one reconciliation format, the best volume pricing and one outage away from zero revenue. Multiple PSPs buy availability, authorization-rate optimization and negotiating leverage, at the cost of a routing layer, N reconciliation pipelines and the cross-PSP duplicate risk.

Staff default: "Single PSP until the math says otherwise. When we add a second, it's active-passive for new intents only, with stored cards made portable via network tokens or a PCI-compliant vault. UNKNOWN intents are pinned to their original PSP forever."

When to deviate:

• GMV where a 4-hour outage costs more than a team-year — roughly $20–50M/day for most businesses — build active-passive.
• Global businesses — local acquiring in the EU, India or Brazil raises authorization rates materially; multi-PSP becomes a revenue lever, not just availability.

🧭 Principal Move: "Portability of stored cards is the one-way door here. If we let the PSP own the vault, switching later means a card-data migration between PCI Level 1 providers — typically months, coordinated with both vendors. I'd negotiate data-portability terms in the contract before signing, not after."

3.5 Fault Line 5: Fraud/Risk — Fail-Open vs Fail-Closed#

The tension: When the risk engine times out or is down, you either approve without a score (conversion preserved, fraud exposure) or decline/hold (fraud prevented, legitimate sales lost). Neither engineering nor product can make this call alone — it's a loss-budget decision.

Staff default: "Degraded rules plus hold-capture. Risk scoring has a 150ms timeout; on timeout, a local rule set runs — amount under $200, known device, card previously used successfully — approve; otherwise authorize but don't capture until re-scored. The loss budget for fail-open windows is set by risk and signed by finance, in dollars per hour."

When to deviate:

• High-risk verticals (gift cards, electronics, crypto on-ramps): fail-closed. The fraud loss rate on unscored gift cards can exceed the margin on the entire category.
• Subscription renewals of long-tenured customers: fail-open. Fraud on a 3-year-old subscription renewal is near zero.

🧭 Principal Move: "I'd publish a risk posture table per product line — fail-open, degraded or fail-closed — signed by the GM and the head of risk. Engineering implements the table; engineering does not decide it. When an outage hits at 2am, the on-call executes the table instead of guessing."

4. Failure Modes & Operational Reality#

4.1 PSP Brownout — The UNKNOWN Flood#

t=0:       PSP A p99 rises from 3s to 25s. Error rate 0.2% → 4%.
t=+30s:    Our 15s timeout starts firing. payment.unknown_count: 12 → 340.
t=+1min:   Checkout threads saturate waiting on PSP. Gateway p99 → 16s.
t=+2min:   Resolver kicks in at full speed → doubles call volume to the struggling PSP.
t=+3min:   PSP A rate-limits us (429). UNKNOWN count 2,100 and climbing.
t=+5min:   Page: payment.unknown_count > 500 for 3m. Also psp.error_rate > 2%.
t=+8min:   On-call: circuit-open PSP A for NEW intents → route to PSP B (portable tokens only).
t=+10min:  Resolver throttled to 20 req/s against PSP A. UNKNOWNs pinned.
t=+45min:  PSP A recovers. Resolver drains 2,100 UNKNOWNs in ~2 min: 1,870 AUTHORIZED, 230 DECLINED.
t=+1 day:  Reconciliation: 3 breaks (webhook arrived after resolver; dedup handled 2, 1 manual).

Detection: psp.auth_latency_p99, psp.error_rate, payment.unknown_count, payment.unknown_age_p99, checkout.thread_pool_utilization.

Mitigation: bulkhead PSP calls into a dedicated pool (e.g., 200 concurrent per PSP) so a slow PSP can't take checkout threads; adaptive resolver rate (token bucket, 20/s default); circuit breaker on the router for new intents.

Prevention: quarterly failover drills; a resolver that respects Retry-After; contractually agreed PSP status notifications.

Owner: payments on-call (primary), with PSP account manager escalation path in the runbook.

4.2 The Cross-PSP Double Charge#

t=0:       Intent pi_9 sent to PSP A. Timeout. State UNKNOWN.
t=+5s:     Router sees PSP A unhealthy. A "retry on failure" branch re-routes pi_9 to PSP B.
t=+6s:     PSP B authorizes pi_9. State AUTHORIZED (psp=B).
t=+40min:  PSP A recovers; webhook: pi_9 AUTHORIZED at PSP A.
t=+40min:  Webhook handler: intent is already AUTHORIZED → ignores it (dedup by intent).
t=+3 days: PSP A auth hold expires silently. Customer saw a pending hold for 3 days.
           Worse variant: auto-capture was on at PSP A → customer charged twice.

Detection: webhook.conflict_total (webhook for an intent that's terminal on a different PSP) — this must page, not log. Reconciliation break type missing_in_ledger for PSP A.

Mitigation: void/refund the orphan at PSP A immediately; notify the customer proactively.

Prevention: the pin rule is enforced in the router, not by convention — psp on the intent is write-once after the first attempt row exists. Code review checklist item and a property test.

Owner: payments team; incident review with the router's author.

4.3 Ledger Imbalance — The Silent Sev-1#

t=0:       Deploy adds FX fee handling. A code path posts the fee debit but not its credit
           when currency != USD (0.4% of volume).
t=+1h:     ledger.imbalance_total = -$212. No alert — the check runs nightly.
t=+14h:    Nightly invariant job: sum(debits) != sum(credits) by $3,090. Page.
t=+15h:    Revert. 1,140 unbalanced transactions identified by txn_id.
t=+2 days: Correcting entries posted, each referencing the incident. Auditor notified.

Detection: make imbalance impossible at write time: the ledger API accepts a transaction (list of entries) only if debits = credits per currency; reject otherwise. Then ledger.rejected_unbalanced_total pages on the first occurrence, within seconds.

Mitigation: correcting entries, never UPDATE/DELETE.

Prevention: the ledger service is the only writer; posting rules live in versioned config reviewed by finance; property tests on every posting rule.

Owner: ledger team (platform); finance-ops signs off on correcting entries.

4.4 Lost Events — The Dual-Write Gap#

t=0:       Service commits intent CAPTURED, then publishes to Kafka.
t=+1ms:    Pod OOM-killed between commit and publish.
t=+1 day:  Ledger has no capture entry. Fulfillment never shipped. Customer charged.
t=+3 days: Reconciliation flags missing_in_ledger. Support ticket already open.

Detection: outbox.lag_seconds and outbox.unpublished_count if using an outbox; if not, only reconciliation finds it — days late.

Prevention: transactional outbox (Appendix C). This is not optional in payments.

Owner: payments team.

4.5 Webhook Storm and Out-of-Order Delivery#

PSP webhooks are at-least-once and unordered. After a PSP incident, you can receive charge.refunded before charge.captured, or 50,000 retried webhooks in 10 minutes.

Mitigation: acknowledge fast (write raw event to a table keyed by PSP event ID, return 200 in < 100ms), process async; state transitions use conditional updates so an out-of-order event either applies (valid transition) or parks for retry; never trust webhook payloads for amounts without fetching the object from the PSP API when stakes are high.

Detection: webhook.ingest_lag_seconds, webhook.parked_count, webhook.signature_failures_total (spoofing attempts).

Owner: payments team.

4.6 Operational Reality Matrix#

🎯 Staff Insight: The idempotency store is the one component that must fail closed. Processing a charge without checking the key trades a few minutes of 503s for an unbounded number of duplicates. "I'd rather take a 5-minute checkout outage than a 5-minute double-charge window."

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Deep card-network knowledge ≠ payment system design. Explaining interchange, BIN ranges and ISO 8583 fields impresses briefly; if it doesn't lead to the UNKNOWN state, it's trivia.
• Saga vocabulary ≠ correctness. "We'll use a saga with compensations" is only Staff if the candidate names what the compensation is for a charge (void vs refund), what it costs, and what happens when the compensation itself times out.
• Microservice count ≠ maturity. Twelve services (wallet, ledger, fraud, router, vault, notifier…) drawn in 5 minutes usually means none of their contracts were thought through.
• PCI buzzwords ≠ compliance design. "We'll be PCI compliant" is not a design; "the PAN never touches our servers because the PSP SDK tokenizes in the browser" is.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Card-network internals (issuer/acquirer message flows) — one sentence, then abstract behind the PSP.
• UI details of checkout — except "Confirming" state for UNKNOWN.
• Currency conversion math — say "FX is a ledger transaction with a rate reference"; skip rate sourcing.
• Tax calculation — separate system; mention it exists.
• Sharding the payments DB — unless the interviewer insists on 100K+ TPS; say why you're skipping it.

6.4 Follow-Up Questions to Expect#

1. "Who generates the idempotency key, and what happens if two browser tabs submit the same order?"
2. "The service crashed after the PSP responded but before you wrote the result. Walk me through recovery."
3. "How do you handle a partial refund followed by a chargeback on the same payment?"
4. "Finance says the PSP payout was $1,204 short yesterday. How do you find it?"
5. "How long do you keep idempotency keys, and what happens when a retry arrives after expiry?"
6. "The risk engine is down. What does checkout do, and who decided?"
7. "How would you migrate from PSP A to PSP B with 4 million stored cards?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design a payment system for our e-commerce site."

Drill 2: Idempotency Key Design#

Prompt: "Walk me through your idempotency keys. Who creates them, what's their scope, how long do they live?"

Drill 3: Crash After PSP Success#

Prompt: "The PSP returned 'authorized'. Your service crashed before writing it. What happens?"

Drill 4: PSP Is Down#

Prompt: "Your primary PSP is returning errors for 40% of calls. What happens?"

Drill 5: Hot Ledger Account#

Prompt: "Every payment credits one platform_revenue account. At 2,000 TPS your ledger writes contend on that row. Fix it."

Drill 6: Marketplace Payouts#

Prompt: "We're becoming a marketplace. Sellers get paid. What changes?"

Drill 7: Build vs Buy#

Prompt: "Why not just use Stripe for everything and skip your own ledger?"

Drill 8: Changing Posting Rules Without an Outage#

Prompt: "Finance wants fees recognized differently starting next month. How do you change ledger behavior safely?"

Drill 9: Multi-Region#

Prompt: "We're expanding to the EU. Make payments multi-region."

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Incident — Flash Sale Double Charges#

Context: During a flash sale at 40× normal traffic, support receives 300 "charged twice" tickets in an hour. The on-call escalates to you.

Questions to Surface First:

• Are these two captures, or an auth hold plus a capture (which looks like two charges on a banking app)?
• Do the duplicates share an order_id? An idempotency key? A PSP?
• Did anything deploy in the last 24 hours — client, router, PSP SDK?
• What's payment.unknown_count doing?

Typical L5 Approach: Finds the duplicates, refunds them, adds a check "if order already paid, reject". Fixes the symptom, doesn't find why the key didn't protect.

Staff Approach: First establishes whether they're real duplicates (most "double charge" reports during high load are pending auth holds plus the capture). Then traces a sample: two intents for one order means the order-level unique constraint is missing or bypassed; one intent with two PSP charges means the PSP key changed between retries.

Principal Approach: Treats it as a control failure: the idempotency guarantee should be structurally impossible to bypass. Commissions a cross-team review of every PSP call site, and moves idempotency into the single PSP client with egress restrictions, so the next team can't reintroduce the bug.

Key Takeaway: "Idempotency keys protect retries of one intent. Only a business-level uniqueness constraint protects against two intents."

What clears the Staff bar:

• Distinguishes auth holds from real duplicates before acting
• Finds the missing business-level constraint, not just the bad key
• Moves correctness ownership to the payments SDK

Deep Dive 2: Silent Failure — Webhooks Stopped Three Days Ago#

Context: Finance notices captured payments from the last 3 days have no settlement matches for one PSP. No alerts fired. Orders are stuck in AUTHORIZED; fulfillment shipped nothing.

Questions to Surface First:

• Did captures reach the PSP, or did our capture calls fail silently?
• Did webhook ingestion stop, or did processing stop?
• Why didn't payment.unknown_count or state-age metrics fire?

Typical L5 Approach: Finds the webhook endpoint was returning 401 after a secret rotation, fixes the secret, replays webhooks from the PSP dashboard.

Staff Approach: Fixes the secret, then fixes the detection gap: the system relied on webhooks as the only path to advance state. Adds the resolver as a second path for any intent in a non-terminal state older than its expected dwell time, plus an alert on state_age_p99 per state.

Principal Approach: Institutes "dwell-time SLOs" for every state in every money state machine as a platform standard, and moves secret rotation for PSP webhooks to an automated dual-secret process owned by the security platform.

Key Takeaway: "Push is an optimization. Pull is the guarantee."

What clears the Staff bar:

• Treats webhooks as at-least-once hints with a pull fallback
• Alerts on state dwell time, not only on errors
• Reconciliation framed as the last line, not the first

Deep Dive 3: Large-Customer Onboarding — Enterprise Invoices at $2M Each#

Context: Sales signs an enterprise buyer paying by bank transfer, $2M per invoice, 20 per month. Current system is card-only with 30-minute UNKNOWN resolution.

Questions to Surface First:

• Push (customer sends a wire) or pull (ACH debit)? Pull debits can be returned for days.
• How do we match an incoming wire to an invoice when the reference is free text?
• What's the approval flow for refunding $2M?

Typical L5 Approach: Adds a "bank transfer" payment method type to the existing flow.

Staff Approach: Recognizes a different state machine: AWAITING_FUNDS → FUNDS_RECEIVED → MATCHED → APPLIED, with a virtual account number per customer so matching is deterministic. ACH returns can arrive up to ~2 banking days (longer for some return codes), so "collected" is not final at receipt. Large refunds require dual approval (four-eyes) enforced in the refund API.

Principal Approach: Separates "payments" (card rails) from "treasury operations" (bank rails, cash positioning) as distinct capabilities with distinct owners, and sets thresholds above which money movement requires finance approval regardless of rail.

Key Takeaway: "A new rail is a new state machine. Don't bolt it onto card semantics."

What clears the Staff bar:

• Recognizes different finality semantics per rail
• Designs deterministic matching instead of fuzzy text matching
• Adds approval controls proportional to amount

Deep Dive 4: Post-Mortem — $180K Reconciliation Hole at Quarter-End#

Context: Quarter-end close finds $180K of captures in the ledger with no PSP settlement. Auditors are asking. You own the post-mortem.

Questions to Surface First:

• Were these captures ever sent? Accepted by the PSP? Later reversed?
• Is the gap concentrated by date, currency, merchant category or code path?
• How long have the breaks existed, and why did nobody act?

Typical L5 Approach: Traces the specific transactions, finds a bug in partial-capture handling (ledger posted full amount, PSP captured partial), fixes the code, posts corrections.

Staff Approach: Fixes the bug, and then asks why breaks aged 70 days without action: the break queue had no owner after a reorg, and the dashboard showed counts, not dollars. Assigns an owner, adds dollar-weighted aging alerts, and adds a posting-rule property test that ledger amount = PSP-confirmed amount.

Principal Approach: Treats it as a governance gap: reconciliation had no named owner at the org-chart level. Writes reconciliation into the finance operating model — owner, SLA, escalation to controller — and makes "unreconciled $ > 30 days" a quarterly audit committee metric.

Key Takeaway: "Reconciliation without an owner is a report nobody reads."

What clears the Staff bar:

• Weights breaks by dollars and age, not count
• Finds the org gap behind the code bug
• Uses correcting entries, never edits

Deep Dive 5: Multi-Region Expansion — India Launch#

Context: The company launches in India. Local regulation requires payment data to be stored in-country and card-on-file flows use tokenization mandated by the regulator; recurring payments need additional mandate registration.

Questions to Surface First:

• What data must stay in-country, and can derived/aggregated data leave?
• Which PSPs have local acquiring and support the mandated token flows?
• What's our legal entity structure in-country?

Typical L5 Approach: Deploys the existing stack in an India region.

Staff Approach: Deploys a regional payments cell with its own DB and ledger for the Indian entity, a local PSP, and a mandate-aware subscription flow with its own states (mandate pending, mandate active, pre-debit notification sent). Global reporting consumes aggregated, non-restricted ledger summaries.

Principal Approach: Establishes the "payments cell per regulatory jurisdiction" pattern as the paved road — deployable template, entity-scoped ledger, local PSP adapter interface — so the next country is a 2-month launch, not a 9-month project, and puts regulatory review into the launch checklist.

Key Takeaway: "Regions follow legal entities, not latency."

What clears the Staff bar:

• Maps regulation to data classification before architecture
• Entity-scoped ledgers with consolidated reporting
• Adapter interfaces so PSPs are replaceable per market

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Explain why exactly-once money movement is unavailable and build exactly-once effect from idempotency, a state machine and reconciliation
• Design idempotency keys at two layers (client → us, us → PSP) with scope, derivation and retention justified
• Draw the payment state machine including UNKNOWN, and state which transitions are conditional updates
• Post a capture, fee, refund and chargeback as balanced double-entry transactions
• Describe a three-way reconciliation with break types, owners, SLAs and dollar-weighted aging
• Handle a PSP brownout with separate behavior for new, unknown and in-flight work
• Name the one store that must fail closed (idempotency) and the decision that engineering must not own alone (fraud fail-open)
• Price multi-PSP routing against outage cost, and say when you would not build it

The Bar for This Question#

Mid-level (L4): Integrates a PSP correctly on the happy path: tokenization via SDK, a charge call, a payments table, webhook handling. Mentions retries. Doesn't distinguish timeout from failure. Would ship something that works 99.9% of the time and double-charges in the remaining 0.1%.

Senior (L5): Adds idempotency keys, two-step auth/capture, an event bus and reasonable failure handling. Knows reconciliation exists. The gap: treats the idempotency key as a server-side dedup detail rather than a contract; keeps money in a mutable status table; proposes PSP failover without pinning UNKNOWNs. The design is plausible and would pass a code review — and would still produce a quarter-end reconciliation hole.

Staff+ (L6): Frames the problem around unknown outcomes within the first five minutes. Separates state from ledger. Uses pre-call/call/post-call phases with no transaction across the network. Designs reconciliation as an owned product with break types, SLAs and dollar-weighted aging. Names who pays for each tradeoff — customer uncertainty over duplicate charges, finance-ops for breaks, risk and finance for fraud posture. Knows when not to build the ledger. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Exactly-Once Payments" Is a Marketing Term#

The Staff position: Say "exactly-once effect" and list its three components. Anything else is a promise you can't audit.

Why this matters in interviews: Interviewers use "exactly-once" as a trap. Accepting the phrase is a Senior signal; decomposing it is a Staff-level signal.

10.2 Most Companies Should Not Build a Ledger Yet#

The Staff position: The ledger is correct and expensive. Build the event schema that makes it cheap to add, and add it when a named trigger fires.

Why this matters in interviews: Candidates who always build the ledger look dogmatic. Candidates who never build it look inexperienced. Name the trigger.

10.3 The Idempotency Store Should Fail Closed — Even Though Everything Else Fails Open#

The Staff position: A 5-minute checkout outage costs revenue you can mostly recover (users retry). A 5-minute duplicate window costs chargebacks, refunds and trust you can't.

Why this matters in interviews: Most availability-trained engineers default to fail-open. Knowing where not to is the signal.

10.4 Multi-PSP Is Usually a Cost Project Disguised as a Reliability Project#

The Staff position: Justify the second PSP with auth-rate and fees, not uptime. Availability is the bonus.

Why this matters in interviews: It shows you think about payments as a revenue system, not just an integration.

10.5 Reconciliation Is the Most Important Service Nobody Wants to Own#

The Staff position: Every other mechanism in this case study leaks. Reconciliation is where leaks become visible. If it lacks a named owner with an SLA, the company is flying blind on its own cash.

Why this matters in interviews: Ending with reconciliation ownership is the single most Staff-sounding close available for this question.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

The Staff engineer designs a correct payment system. The Principal engineer notices that the company has four payment systems — checkout, subscriptions, the marketplace team's payouts, and a gift-card service someone wrote in 2021 — each with its own idempotency conventions, its own PSP contract and its own definition of "paid". The L7 problem is not correctness of one flow; it is money-movement governance across the org: one ledger of record, one set of PSP contracts, one resolver framework, and a finance-engineering operating model that survives reorgs.

The Org-Level Fault Line#

Central money-movement platform vs per-product PSP integrations.

🧭 Principal Move: "Platform owns the primitives that must be correct once — intents, idempotency, PSP adapters, ledger posting, reconciliation. Products own the flows that must be fast to change — checkout UX, subscription logic, promotions. Direct PSP egress from product services gets blocked after a 2-quarter migration window."

Cost Model#

Assumptions: US-centric, card-heavy, fully loaded engineer cost ~$250K/year, PSP fees at list minus negotiated discounts, cloud infra only for the payments stack.

The pricing insight: at enterprise scale, a routing team of 5 engineers (~$1.25M/yr) that improves blended cost by 5 bps saves ~$5M/yr. At growth scale, the same team saves ~$250K — not worth it yet.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-PAY-001: Money Movement Standard
Status: Approved   Owners: Payments Platform + Office of the Controller

Scope
  Every service that initiates, reverses or records movement of funds
  (card, bank, wallet, payout, stored value).

MUST
  1. Reach PSPs and banks only through the Money Movement API or approved SDK.
  2. Create a durable intent before any external call; carry an idempotency key
     derived from the intent and operation.
  3. Treat timeouts and 5xx as UNKNOWN; never mark FAILED without a definitive response.
  4. Post all financial effects to the company ledger as balanced transactions
     (debits = credits per currency) in integer minor units.
  5. Never store PANs; use approved tokenization.
  6. Define dwell-time SLOs for every non-terminal state.

SHOULD
  1. Use two-step authorize/capture for goods with fulfillment delay.
  2. Emit events via transactional outbox.
  3. Participate in daily reconciliation with named break owners.

Exceptions
  Filed with Payments Platform; reviewed within 10 business days; time-boxed to
  2 quarters; controller sign-off required for any exception to MUST 4.

Success metrics
  - Duplicate captures per million payments: < 1
  - UNKNOWNs resolved within 1h: ≥ 99.9%
  - Reconciliation break rate: < 0.01%; breaks > 30 days: $0
  - Month-end close: ≤ 3 business days
  - Services with direct PSP egress: 0 by end of year 2

What I'd Tell the VP#

"Our payment systems are correct individually but inconsistent collectively — four teams move money four different ways, and finance spends eight days a month reconciling them. I'm proposing a money-movement platform that owns the parts that must be right once: idempotency, the ledger and reconciliation. It costs about six engineers for a year and cuts close time to three days, removes the double-charge class of incidents, and gives us one negotiating position with PSPs worth an estimated 10–20 basis points. Product teams keep ownership of checkout and pricing. The main risk is migration speed; I'd time-box direct PSP integrations to two quarters."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll build a great payment service for checkout" — correct, but ignores the three other teams moving money differently.
• "Reconciliation runs nightly and finance handles breaks" — names the owner but not the operating model, escalation path or audit metric.
• "We'll add a second PSP for resilience" — no cost model, no mention of token portability as the one-way door.

Appendices

Appendix A: Mechanics in Depth#

A.1 The Three-Phase Request (Pre-call / Call / Post-call)#

def confirm(intent_id, client_key, request):
    # Phase 1: local transaction — record intent to act
    with db.tx() as tx:
        rec = tx.idem_get_for_update(scope="confirm", key=client_key)
        if rec and rec.request_hash != hash(request): raise Http422
        if rec and rec.status == "DONE":         return rec.response
        if rec and rec.status == "IN_PROGRESS":  return Http202("PROCESSING")
        tx.idem_insert(scope="confirm", key=client_key, hash=hash(request), status="IN_PROGRESS")
        intent = tx.intent_get_for_update(intent_id)
        attempt = tx.attempt_insert(intent_id, op="authorize",
                                    psp_key=f"{intent_id}:authorize:{intent.attempt_seq+1}",
                                    outcome="PENDING")
    # Phase 2: network call — no DB transaction open
    try:
        result = psp.authorize(intent, idempotency_key=attempt.psp_key, timeout=15s)
    except (Timeout, Psp5xx):
        result = UNKNOWN
    # Phase 3: local transaction — record outcome
    with db.tx() as tx:
        tx.attempt_update(attempt.id, outcome=result.outcome, psp_ref=result.ref)
        new_state = {"SUCCEEDED": "AUTHORIZED", "DECLINED": "DECLINED", "UNKNOWN": "UNKNOWN"}[result.outcome]
        tx.intent_transition(intent_id, from_state="AUTHORIZING", to_state=new_state)  # conditional
        tx.outbox_insert(event=f"payment.{new_state.lower()}", intent_id=intent_id)
        tx.idem_complete(scope="confirm", key=client_key, response=render(new_state))
    return render(new_state)

A.2 The Resolver#

every 30s:
  for attempt in attempts where outcome in ("PENDING","UNKNOWN")
                           and updated_at < now() - backoff(attempt.tries)
                           limit 500:
      if not psp_budget[attempt.psp].take(): continue      # 20 req/s per PSP
      r = psp.lookup(attempt.psp_ref or attempt.psp_key) or psp.retry(attempt, same key)
      if r.definitive: phase3(attempt, r)
      elif attempt.age > 24h: human_queue.push(attempt)
backoff: 1m, 5m, 30m, 2h, 6h

A.3 Why Not Sagas With Automatic Compensation?#

A saga compensates a failed step by undoing earlier ones. For payments, "undo" is void (cheap, pre-capture) or refund (expensive, customer-visible, 5–10 days). Automatic compensation on UNKNOWN is wrong — you'd refund a charge that may not exist, or void an auth that's about to be captured. Use sagas for the order workflow (reserve inventory → authorize → capture on ship → release inventory on decline) with payment steps that only compensate on definitive outcomes. See Managing Long-Running Processes.

Appendix B: Data Model#

CREATE TABLE payment_intents (
  intent_id      TEXT PRIMARY KEY,
  order_id       TEXT NOT NULL,
  amount_minor   BIGINT NOT NULL CHECK (amount_minor > 0),
  currency       CHAR(3) NOT NULL,
  state          TEXT NOT NULL,
  psp            TEXT,                         -- write-once after first attempt
  attempt_seq    INT NOT NULL DEFAULT 0,
  version        BIGINT NOT NULL DEFAULT 0,
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX one_live_intent_per_order ON payment_intents(order_id)
  WHERE state NOT IN ('DECLINED','VOIDED','EXPIRED');

CREATE TABLE ledger_entries (
  entry_id       BIGSERIAL PRIMARY KEY,
  txn_id         UUID NOT NULL,
  account_id     TEXT NOT NULL,
  direction      CHAR(1) NOT NULL CHECK (direction IN ('D','C')),
  amount_minor   BIGINT NOT NULL CHECK (amount_minor > 0),
  currency       CHAR(3) NOT NULL,
  caused_by      TEXT NOT NULL,                -- event id; unique per (caused_by, account, direction)
  rule_version   INT NOT NULL,
  posted_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- No UPDATE/DELETE grants on ledger_entries for any application role.

The ledger API accepts a transaction only when Σ debits = Σ credits per currency; the check runs inside the insert transaction. UNIQUE(caused_by, account_id, direction) makes posting idempotent: a replayed event cannot double-post.

Appendix C: Coordination Mechanisms#

C.1 Transactional Outbox#

C.2 Quick Comparison#

Appendix D: API Contract & Client Behavior#

• Idempotency-Key required on every POST; 400 if absent.
• Replay with same key + same body → original status and body; replay with different body → 422 idempotency_key_reused.
• 202 PROCESSING for in-progress or UNKNOWN; include Retry-After: 2 and a status URL.
• Clients poll with exponential backoff capped at 30s, or subscribe to push; clients never create a new intent for the same order after a timeout.
• Webhooks to merchants/consumers are signed (HMAC with timestamp, 5-minute tolerance), carry a stable event_id, and are retried with backoff for up to 3 days.

Appendix E: Observability#

Core metrics:

• payment.unknown_count, payment.unknown_age_p99, money_in_unknown_usd
• payment.state_age_p99{state} — dwell time per state
• psp.auth_latency_p99{psp}, psp.error_rate{psp}, payment.auth_rate{psp,bin_country}
• ledger.rejected_unbalanced_total (must stay 0), ledger.post_lag_seconds
• outbox.unpublished_age_max
• recon.break_rate_pct, recon.break_usd_by_age_bucket
• payment.duplicate_capture_per_order (must stay 0)

Critical alerts:

Debugging the silent failure: a drop in authorization rate with no error-rate change is the most common silent failure (issuer-side declines after a PSP configuration change, a 3DS flow break, a BIN-country rule). Alert on auth rate by BIN country and PSP, not only on errors.

Appendix F: Scale Evolution#

What you don't build on day one: multi-PSP routing, your own vault, multi-region active-active anything, a custom ledger database, real-time reconciliation. Each has a trigger in Section 11.

Appendix G: Multi-Tenancy, Fairness & Cost#

• PSP rate limits are shared. A batch subscription renewal of 2M charges at 00:00 UTC can consume the PSP's per-account rate limit and starve interactive checkout. Separate PSP accounts or priority lanes: interactive traffic gets ≥ 70% of the PSP budget; batch renewals are spread over 6 hours.
• Per-merchant isolation (marketplaces/platforms): one merchant's chargeback storm must not raise the platform's overall dispute ratio unnoticed — track dispute ratio per merchant and per MCC, with automated reserves at 0.75%.
• Cost attribution: tag every ledger fee entry with product line so PSP fees are charged back; product teams that pick expensive payment methods see the cost.
• Retries cost money: some PSPs and networks charge per authorization attempt, and networks penalize excessive retries on declined cards. Cap retries on hard declines at zero and soft declines at a small, network-compliant number.