Technologies referenced in this case study: Kafka · PostgreSQL · ZooKeeper & etcd · Time Series DBs

Related: Distributed Consensus · Payment Processing · Dealing with Contention · Real-time Updates · Network Latency & Protocols · Replicated Data Store

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once, then return to individual sections before a loop.

Without a sequencer and deterministic replay:
t=0:        Two matching threads share a symbol's book for throughput
t=+2ms:     Buy 500 @ 100.05 and Sell 500 @ 100.05 arrive 3µs apart on different gateways
t=+2ms:     Thread A matches the buy against a resting sell; thread B matches the same
            resting sell against another order. Lock ordering bug: 500 shares filled twice.
t=+3ms:     Two execution reports go out. Market data prints both trades.
t=+40min:   Primary crashes. Backup rebuilds from logs in a different interleaving;
            its book disagrees with what members were told.
t=+1 day:   Trades busted. Regulator inquiry. Members demand compensation.

With a sequencer + single-threaded deterministic engine:
t=0:        Every inbound message gets a global sequence number and is journaled
t=+2ms:     Engine for the symbol processes them strictly in sequence order, one at a time
t=+2ms:     Exactly one fill. Execution reports and market data carry the sequence number.
t=+40min:   Primary crashes. Hot standby has processed the identical sequence;
            its book hash matches. Failover in < 1s. No bust, no inquiry.

Executive Summary

If you only read one section, read this. Every decision in the case study follows from the contrast and the fault lines below.

What This Interview Actually Tests#

A stock exchange is not a throughput question. Matching 1 million orders a second on one core was demonstrated publicly by LMAX more than a decade ago.

It is a determinism and fairness question that tests:

• Whether you can defend a single-threaded design against the reflex to parallelize
• Whether you establish one total order of events and make everything downstream a pure function of it
• Whether recovery is "replay the journal and get the identical state" rather than "restore from a database"
• Whether you treat fairness — equal access, equal information, provable ordering — as the product
• Whether you know when the right move is to halt trading

The key insight: An exchange is a replicated state machine whose input log is legally significant. If two replicas given the same log can ever disagree, you don't have an exchange; you have a liability.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll design a regulated lit exchange with continuous price-time matching and opening/closing auctions, because that's where determinism, fairness and latency collide hardest. I'll target under 50µs from order receipt to acknowledgment inside our network, and I'll treat the inbound journal as the legal record."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: These are public, well-documented designs. Citing them shows you know the single-threaded model is production reality, not a toy.

LMAX Exchange — The Disruptor and a Single Business-Logic Thread#

LMAX publicly described (via Martin Fowler's well-known write-up) an architecture where all business logic for its retail FX exchange ran on a single thread, processing on the order of 6 million orders per second in memory. Input events were journaled and replicated before processing via the Disruptor, a lock-free ring buffer; recovery came from snapshots plus event replay.

Staff insight: The canonical proof that "single-threaded" is a performance strategy, not a limitation. The reason it's fast is the same reason it's correct: no locks, no contention, cache-friendly sequential processing, and every state change is a deterministic function of the input log.

Nasdaq — INET, OUCH and ITCH#

Nasdaq's matching technology (derived from INET) is paired with published protocols: OUCH for order entry and ITCH for market data — a binary, sequence-numbered feed disseminated over UDP multicast (with the MoldUDP64 framing and a retransmission service for gaps). The same message types that describe book changes let any participant rebuild the full order book.

Staff insight: The market data feed is itself a replicated log. Participants are replicas of the book; sequence numbers plus a retransmit service turn unreliable multicast into a reliable, fair, one-to-many log.

IEX — The Speed Bump#

IEX, which became a registered US exchange in 2016, famously introduced a 350-microsecond delay implemented with about 38 miles of coiled fiber on inbound and outbound paths, so that its own price-sensitive order types could react to market changes before fast traders could exploit stale quotes.

Staff insight: Fairness is a design parameter, not an afterthought. IEX deliberately traded latency for a fairness property — and needed regulatory approval to do it. When an interviewer asks "how do you make it faster?", a Staff answer includes "faster for whom, and is that the product?"

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: Orders pass session checks and pre-trade risk in the gateway, then get a global sequence number from the sequencer, which replicates the message to at least two journal hosts in memory before releasing it. Matching engines — one thread per symbol partition — consume the sequenced stream; a hot standby consumes the identical stream. Outputs (acks, fills, book updates) are themselves sequenced and published: execution reports on the member's session, market data on multicast with a retransmit service. Market operations can halt or trigger auctions by injecting sequenced control messages, so even halts are replayable.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates design a web service — REST API, load balancer, stateless workers, a database of orders — and spend 20 minutes on it. Then the interviewer asks "what if two orders arrive at the same time?" and the design has no answer. Establish the sequencer and the single-threaded engine in the first 10 minutes.

Phase 1: Requirements & Framing (2–3 minutes)#

Functional scope in one sentence:

"Members submit, modify and cancel limit and market orders; we match them with price-time priority, send acknowledgments and fills, and publish order-book and trade data to the market."

Then the non-functional contract — this is the Staff investment:

"Four questions: What market structure — continuous lit trading, auctions, or both? What latency class — are members co-located high-frequency firms or retail over the internet? What regulatory regime — do we need to prove order handling to a regulator? And what's the failure policy — is a halt acceptable? I'll assume a regulated lit equities venue, co-located members, a requirement to reproduce any trade from the audit record, and that halting a symbol is acceptable when correctness is in doubt."

Commit to numbers:

"Targets: order-to-ack under 50µs p99 inside our network, 10K symbols, peak 2M inbound messages/sec across the venue, the busiest symbol at ~200K messages/sec, zero system-caused busted trades, failover under 1 second, market data published to all subscribers within the same microsecond window."

🎯 Staff Move: "I'm going to be opinionated about one thing up front: correctness beats availability here. If the engine's state is ever in doubt, we halt the symbol. A wrong trade is a legal event; a halt is an operational event."

Phase 2: Core Entities & API (1–2 minutes)#

• Order: order_id, member_id, symbol, side, price (integer ticks — never floats), qty, tif (DAY/IOC/FOK), seq
• Order Book: per symbol, bids and asks, each a sorted map of price level → FIFO queue of orders
• Execution / Trade: trade_id, buy_order, sell_order, price, qty, seq
• Sequenced Message: seq, type, payload, gateway_ts — the unit of the journal

Order entry (binary protocol over TCP session, OUCH-like; FIX for slower members):

EnterOrder(client_order_token, symbol, side, price_ticks, qty, tif)  → Accepted(seq, order_id) | Rejected(reason)
CancelOrder(order_id, qty_to_cancel)                                   → Canceled(seq)
ReplaceOrder(order_id, new_price, new_qty)                             → Replaced(seq, new_order_id)   # loses time priority if price changes or qty increases
Outbound: Executed(seq, order_id, qty, price, trade_id)

🎯 Staff Move: "Prices are integers in ticks. A floating-point price in a matching engine is a nondeterminism bug waiting to happen — two machines rounding differently is a divergent book."

Phase 3: High-Level Architecture (≤5 minutes)#

Walk the flow in 90 seconds:

1. Member's order hits a gateway: session validation, message-rate throttle, pre-trade risk.
2. Gateway forwards to the sequencer, which stamps a monotonically increasing sequence number and replicates the message to at least two hosts' memory.
3. The matching engine for that symbol group consumes messages in sequence order, one at a time, and updates the in-memory book.
4. Outputs — ack, fills, book deltas — are emitted in the same order, tagged with the input sequence number.
5. The hot standby consumes the same stream and produces identical state; outputs are suppressed unless it's promoted.

🎯 Staff Move: "That's the whole core. Everything interesting is about keeping it deterministic, fast and fair. I'd like to go deep on three things: why single-threaded and how we scale it, how recovery and failover work by replay, and how we disseminate market data fairly."

Phase 4: Transition to Depth (1 minute)#

"The architecture is simple on purpose. The fault lines are: determinism — what's allowed to influence the engine; fairness — what 'first' means when two orders are 1µs apart on different gateways; and failure — when we fail over versus halt. Which would you like first?"

If no preference: lead with determinism and replay — it underpins the other two.

Phase 5: Deep Dives (25–30 minutes)#

Deep dive A: Why one thread, and how it scales (6–8 min)

"Matching is a sequence of small, data-dependent operations on a hot, in-memory structure. A single core with the book in L1/L2 cache processes an order in hundreds of nanoseconds to a few microseconds. Adding threads to the same book adds lock acquisition, cache-line bouncing and — worst — nondeterministic interleaving. So: one thread per book. Throughput scales by partitioning symbols across engines: 10K symbols over 16 engine processes on dedicated cores. The busiest symbol at 200K messages/sec uses maybe 20–40% of one core. Partition assignment is by measured volume, not alphabetically, and rebalancing happens overnight, not intraday."

Deep dive B: Sequencer, journal and replay (6–8 min)

"The sequencer is the only place where order is decided. It assigns sequence numbers and doesn't release a message to engines until it's in memory on at least two hosts in different racks — a few microseconds on a low-latency switch. Disk writes are asynchronous and batched. The engine is a pure function: state_n = apply(state_{n-1}, msg_n). It never reads the wall clock — time comes in as sequenced timer messages. Every few minutes we snapshot the book; recovery is 'load snapshot at seq S, replay S+1 through the end'. At 2M msgs/sec and a snapshot every 60s, replay is at most 120M messages — about 20–60 seconds at replay speed, which is why the standby is hot rather than cold."

Deep dive C: Failover versus halt (5–7 min)

"The standby has consumed every sequenced message; it's at the same sequence number as the primary within microseconds. Every 10K messages, both emit a hash of book state onto a side channel. If the primary dies, the standby promotes — within a few hundred ms including detection — and resumes emitting outputs from the next sequence. Members see a brief pause. If the hashes ever mismatch, we halt the affected symbols, because we can't know which book is true. Halt means: stop accepting new orders, keep cancels, publish a halt status on market data, and reopen through an auction once reconciled."

Deep dive D: Fair market data dissemination (4–6 min)

"Book deltas and trades are published as sequenced messages over UDP multicast — one send reaches every subscriber at the switch simultaneously. Two redundant feeds (A and B) over separate networks; subscribers arbitrate by sequence number. Gaps are filled from a retransmit service; late joiners get a snapshot then the incremental feed. In the co-location facility, cross-connects are equal-length so no member is physically closer. Per-client TCP streams would be both unfair — the 1,000th client gets data after the first 999 — and unscalable."

Phase 6: Wrap-Up (2–3 minutes)#

"An exchange is a deterministic state machine behind a sequencer. Fairness comes from a single total order and equal dissemination; resilience comes from replaying that order on a hot standby; correctness is protected by halting rather than guessing. The operational story is market ops owning halts and kill switches, the engine team owning determinism, and a replayable journal that serves recovery, testing and the regulator from the same artifact."

"What I'd build next: closing-auction optimization, a certification environment where members replay their production traffic against new engine builds, and a surveillance pipeline reading the journal."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

The exchange question punishes pattern-matching. Every habit from web-scale design — horizontal scaling, eventual consistency, retries with at-least-once semantics, timestamps for ordering — is wrong in the core. Staff candidates recognize that the problem is a replicated state machine with a legally significant log, and design everything around keeping it deterministic. They also recognize the organizational weight: market operations, compliance, and members are all stakeholders in a failure.

What interviewers listen for:

1. A single total order established before matching
2. A defended choice of single-threaded matching with a scaling story
3. Recovery and replication by replay, with state verification
4. Concrete pre-trade controls and kill switches with a latency budget
5. "Halt" named as a first-class, designed failure mode

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"Two orders for the same symbol arrive 1 microsecond apart on two different gateways. Which one gets filled, how do you prove it to a regulator a year later, and would your backup have made the same decision?"

A Staff answer: the sequencer decides; the journal records the decision with the sequence number; replaying the journal on any engine build reproduces the same fill; the standby made the same decision because it consumed the same sequence.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Regulated lit exchange → determinism, fairness, auditability

• Constraint: order handling must be provable and reproducible; tens of µs latency
• Mechanism: sequencer, single-threaded engines, replicated journal, multicast data
• Failure mode: halt the symbol; reopen via auction
• Who pays: members during a halt (can't trade), market ops (running the reopening), the venue's reputation

Crypto / retail exchange → availability, custody, burst handling

• Constraint: 24×7, no maintenance windows; users hold balances on the platform
• Mechanism: same core engine plus a custody ledger reconciled against the engine; heavy API rate limiting (see Rate Limiting); cancel-only degraded mode
• Failure mode: cancel-only mode rather than halt; withdrawals paused on ledger mismatch
• Who pays: users during cancel-only mode; finance/treasury for reconciliation gaps

Dark pool / periodic auction → information protection, speed neutrality

• Constraint: don't reveal resting interest; latency advantage shouldn't matter
• Mechanism: collect orders for N ms, uncross at a single price; no public book
• Failure mode: skip a batch cycle
• Who pays: participants waiting for the next batch; the operator if leakage occurs

2.2 When NOT to Build a Matching Engine#

• Internal crossing for a small broker. If volume is thousands of orders per day, a database with serializable transactions is simpler and auditable enough.
• Marketplace matching with human timescales (ride-hailing, freelance jobs). These are assignment problems with scoring, not price-time books; see Ride Hailing & Delivery.
• Ticketing / inventory. "First come first served" on seats is a contention problem (Flash Sales, Contention), not an order book.
• When a licensed venue or white-label engine exists. Several vendors sell certified matching engines; building one is justified only if matching behavior is the product differentiator.

2.3 What the Interviewer Leaves Underspecified#

Staff engineers surface these. Senior engineers assume a web app.

2.4 Precise Terminology#

🎯 Staff Insight: If the interviewer says "make it distributed," ask: "Distributed across symbols, or distributed within a single book? The first is how we scale; the second breaks price-time priority and determinism."

3. The Fault Lines#

Each fault line has a technical axis and an organizational one. In an exchange, the organizational axis includes a regulator — which makes "who signed off" a literal question.

3.1 Fault Line 1: Determinism vs Parallelism#

The tension: More threads on a book would raise per-symbol throughput. They also make the book's evolution depend on scheduler timing, which breaks replay, replication, and fairness.

L6 answer: "Parallelism lives around the core — decoding, risk, journaling, market-data encoding each get a core connected by lock-free ring buffers, the LMAX Disruptor pattern. The matching logic itself is one thread. We scale out by symbol partition. I'll forbid anything nondeterministic from entering the core: no wall-clock reads, no random numbers, no iteration over unordered hash maps, no floating point."

When to deviate: Never within a single order book for a regulated venue. For cross-symbol strategies (spread instruments, options with legs), co-locate related books in the same partition so implied matching stays single-threaded.

🧭 Principal Move: "I'd enforce the determinism boundary with tooling, not reviews: a CI job replays yesterday's full production journal against every new build and compares output hashes byte-for-byte. A mismatch blocks the release."

3.2 Fault Line 2: Latency vs Fairness#

The tension: Lower latency is what members pay for. But pure speed rewards whoever spends the most on proximity and hardware, and it lets fast participants pick off stale quotes. Every fairness mechanism costs latency or throughput.

L6 answer: "The sequencer defines fairness — first sequenced, first served — so the fairness work is making arrival at the sequencer fair: equal-length cables in co-location, gateways with identical code paths and latency, and monitoring gateway latency skew across the pool. If a gateway is 3µs slower than its peers, members connected to it are disadvantaged; I'd alert when p99 skew across gateways exceeds 1µs."

When to deviate: If the product strategy is to attract long-term investors over HFT flow, a speed bump or batch auction is a deliberate product choice — made by the business and approved by the regulator, not by engineering.

🎯 Staff Insight: "Gateway latency variance is a fairness bug, not just a performance bug. Two members who send at the same instant should reach the sequencer in the same order regardless of which gateway they're assigned to."

3.3 Fault Line 3: Inline Risk vs Speed#

The tension: Each pre-trade check costs microseconds. Each removed check is a runaway algorithm or fat finger waiting to hit the book.

L6 answer: "All pre-trade state lives in memory in the gateway's risk module, updated from the engine's execution stream so credit usage is current within microseconds. Total budget under 5µs. Plus a per-member kill switch, triggered by the member or market ops, that's itself a sequenced message — so 'all orders canceled at seq N' is replayable."

Who pays: The venue pays latency; the member pays if they disabled their own controls. The Staff move is naming that controls are layered — broker-side (regulatory obligation in many markets), venue-side, and market-wide circuit breakers.

3.4 Fault Line 4: Availability vs Correctness on Failure#

The tension: Trading halts cost members money and cost the venue market share. Trading on a book that may be wrong costs far more — busted trades, lawsuits, regulatory action.

L6 answer: "Fail over when we can prove the standby's book equals the primary's. Halt when we can't. Reopening goes through an auction so the price is re-discovered fairly rather than whoever reconnects first getting the stale quotes."

🧭 Principal Move: "The halt-and-reopen procedure is part of the rulebook filed with the regulator. I'd make sure engineering, market operations and compliance co-own it and rehearse it monthly in the test environment — a halt is where venues lose reputation, and it's almost never practiced."

3.5 Fault Line 5: Market Data — Equal Access vs Scale and Revenue#

The tension: Market data must reach all subscribers simultaneously for fairness. It's also a major revenue line for exchanges, which creates pressure for tiered products (faster, deeper feeds for those who pay more).

L6 answer: "Primary feed is sequenced multicast inside co-location with A/B redundancy, a retransmit service for gaps under N messages, and a snapshot service for late joiners or larger gaps. Retail and cloud consumers get a derived feed via a separate fan-out tier — see Real-Time Updates — explicitly labeled as delayed relative to the direct feed."

🎯 Staff Move: "Market data is a replicated log: every subscriber is a replica of the order book. If they miss a sequence number, their book is wrong until they recover it. That's why the retransmit and snapshot services are part of the core design, not add-ons."

4. Failure Modes & Operational Reality#

4.1 Engine Crash During Peak Trading#

t=0:        09:30:02 — open auction just uncrossed. 1.4M msgs/sec venue-wide.
t=0:        Engine partition 7 (symbols including a top-5 name) segfaults on a new order type.
t=+50µs:    Standby for partition 7 has already processed the same message... and also segfaults.
t=+200ms:   Health monitor: both primary and standby for partition 7 down.
t=+300ms:   Sequencer continues sequencing messages for partition 7; they queue in the journal.
t=+2s:      Market ops halts partition-7 symbols via sequenced HALT message.
t=+4min:    Root cause: new order-type handler dereferences null on an edge case.
t=+25min:   Hotfix disables the order type at the gateway (reject); engines restart,
            replay from snapshot, skip nothing — the offending message is rejected
            deterministically by the new build's validation.
t=+32min:   Reopening auction; trading resumes.

Detection: engine.heartbeat_missing{partition}, engine.last_processed_seq stalled, journal.pending_for_partition growing. Blast radius: All symbols in partition 7 (~600 symbols, ~8% of volume). Key insight: Deterministic replication means a deterministic bug kills the standby too. Hot standby protects against hardware failure, not software bugs. Mitigation: Halt, fix, replay. Gateways can reject the poison message type immediately. Prevention: Journal replay of production traffic in CI; fuzzing new order types; canary new features on low-volume symbols first; feature flags per order type. Owner: Engine team (fix), market operations (halt/reopen), compliance (member notice).

4.2 Sequencer Split Brain#

t=0:       Primary sequencer S1 loses its network link to the coordinator for 800ms.
t=+500ms:  Coordinator lease expires; S2 promoted with epoch 18.
t=+800ms:  S1's link returns. S1 (epoch 17) still has messages in flight.
With fencing:   journal hosts reject epoch 17 writes; S1 steps down. Seqs continuous.
Without fencing: two messages carry sequence 88,120,442 with different content.
                 Engines disagree depending on which they received. Hash mismatch → halt.

Detection: sequencer.epoch_changes, journal.duplicate_seq_rejected, engine state-hash mismatch. Mitigation: Epoch fencing at the journal replicas; lease duration longer than the worst tolerable pause. See Distributed Consensus. Owner: Core infrastructure team.

4.3 Market Data Gap Storm#

t=0:       Microburst: 400K messages in 10ms after a macro news release.
t=+10ms:   Switch buffers overflow on feed A; 2,300 packets dropped.
t=+11ms:   800 subscribers detect gaps; feed B is intact for most — they arbitrate.
t=+12ms:   120 subscribers on a misconfigured host missed both feeds; request retransmit.
t=+15ms:   Retransmit server receives 120 × 2,300 requests; queue grows.
t=+2s:     Subscribers still gapped fall back to snapshot service.

Detection: md.feed_a_gap_rate, md.retransmit_requests_per_sec, md.snapshot_requests. Blast radius: Subscribers trading on stale books — which becomes adverse selection against them. Mitigation: A/B arbitration; retransmit rate limits per subscriber; snapshot fallback. Prevention: Capacity-plan for microbursts (peak per-ms rate, not per-second); conflated feeds for slow consumers. Owner: Market data team.

4.4 Runaway Member Algorithm#

t=0:       Member X deploys a new algo. A flag is misapplied; it sends buy orders in a loop.
t=+1s:     X's session: 45K msgs/sec (normal: 2K). Rate throttle at 10K/sec rejects the excess.
t=+3s:     Orders within the throttle still accumulate: X's net position grows by $8M/sec.
t=+12s:    Credit limit ($100M exposure) reached; gateway rejects all new X orders.
t=+15s:    Market ops sees `member.exposure_utilization{X}` = 100% and calls X.
t=+60s:    X requests kill switch; all X orders canceled at seq N.

Detection: member.msg_rate, member.exposure_utilization, member.reject_rate. Blast radius: Bounded by credit limit; counterparties trade at market prices, not harmed. Mitigation: Throttle, credit limit, kill switch — all pre-agreed with the member. Owner: Member (their algo), market ops (kill switch execution), risk team (limit levels).

4.5 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Knows HFT jargon: Talking about FPGAs and kernel bypass ≠ designing a fair, recoverable venue.
• Order book data-structure depth: A perfect price-level tree with no sequencer is still wrong.
• Mentions Kafka's ordering guarantee: A single-partition log is a valid sequencer shape, but at 5–10ms p99 it's the wrong latency class for co-located trading — the candidate must say so.
• "Blockchain for audit": Tamper-evidence is solvable with hash-chained journals; consensus across untrusted parties isn't the requirement.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

6.4 Follow-Up Questions to Expect#

1. "Your busiest symbol spikes 10×. What happens?"
2. "How do you test a new engine build before production?"
3. "Can the standby ever disagree with the primary? How would you know?"
4. "What does a cancel-replace do to time priority?"
5. "How do you handle the market open, when thousands of orders arrive at once?"
6. "Could we run this in a public cloud?"
7. "How would a regulator reconstruct a trade from last March?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design a stock exchange."

Drill 2: Defend Single-Threaded#

Prompt: "A single thread? That can't possibly scale. Why not a thread pool?"

Drill 3: Make Durability Concrete#

Prompt: "fsync is 50–100µs. Your latency target is 50µs total. How is anything durable?"

Drill 4: The Poison Message#

Prompt: "A new order type crashes the primary engine. What does your hot standby do?"

Drill 5: Hot Symbol#

Prompt: "A meme stock goes from 20K to 400K messages/sec. What happens?"

Drill 6: Multi-Tenant Fairness Among Members#

Prompt: "One large market maker sends 60% of all messages. Other members complain about latency."

Drill 7: Build vs Buy#

Prompt: "Should we build our own matching engine or license one?"

Drill 8: Changing Rules Without an Outage#

Prompt: "We're changing tick sizes for 500 symbols next month. How?"

Drill 9: Cost#

Prompt: "The CFO asks why the exchange infrastructure costs so much."

Drill 10: Multi-Region#

Prompt: "Should we run the matching engine active-active in two regions?"

8. Deep Dive Scenarios#

Deep Dive 1: Peak Traffic — Market Open After Major News#

Context: Overnight news moves the whole market. At the 09:30 open, inbound messages spike to 5× normal peak. Order-to-ack p99 goes from 40µs to 2ms for 90 seconds. Members call market ops. The on-call escalates to you.

Questions to Surface First:

• Where is the queueing — gateways, risk, sequencer, a particular engine partition, or output encoding?
• Is it uniform across partitions or one partition with hot symbols?
• Were any messages rejected or throttled? Any member disproportionately affected?
• Did market data keep pace, or did subscribers trade on stale books?

Typical L5 Approach: Adds more gateway instances and scales up engine hosts for tomorrow. Plausible — but misses that one partition hosting three news-driven symbols saturated while the rest were idle, and adding hosts doesn't change placement.

Staff Approach: Pulls per-stage latency (stage.queue_delay_us for decode, risk, sequencer, engine partition, encode) and finds partition 4's core at 100% for 90s. Rebalances hot symbols to lightly loaded partitions for the next session using overnight volume and news signals, and checks market data lag — found that 3% of subscribers fell back to snapshot, trading blind for ~2s.

Principal Approach: Establishes capacity as a published contract: tested peak per partition and per symbol, pre-open placement driven by a volatility forecast, and quarterly load tests at 10× normal peak replaying real open-auction traffic. Reports p99 at open as a member-facing SLO.

Key Takeaway: "At the open, capacity is a placement problem, not a hardware problem."

What clears the Staff bar:

• Per-stage latency decomposition
• Refuses risky intraday changes
• Placement as the durable fix

Deep Dive 2: Silent Failure — The Standby Was Wrong for Weeks#

Context: A routine planned failover of partition 9 triggers a state-hash mismatch. Investigation shows the standby's book has diverged on a tiny number of orders for three weeks — hashes were computed but the comparison job had been failing silently since a config change.

Questions to Surface First:

• Which book is correct? Can we replay the journal on a clean engine to find out?
• What is the source of nondeterminism?
• Why did the comparison fail silently? Who owns that job?
• Have any live decisions depended on the standby (e.g., earlier failovers in the window)?

Typical L5 Approach: Rebuilds the standby from the primary's snapshot, fixes the comparison job, adds an alert on its failures. Fixes the symptom; doesn't find the nondeterminism.

Staff Approach: Replays the journal on a third, clean engine; finds the primary and replay agree, standby differs. Root cause: the standby host has a different CPU microcode/library build that changed iteration order of an unordered map used in an expiry routine. Removes the unordered iteration (determinism bug), makes hash comparison a heartbeat-level check — missing comparisons page, and blocks failover to a standby without a verified hash in the last 60s.

Principal Approach: Treats "verification job silently failing" as an org-wide class: any safety control that can stop running without anyone noticing needs a dead-man's-switch alert. Standardizes environment parity (pinned builds, identical images) across primary and standby as an audited control, and adds determinism linting (banned constructs) to the engine codebase.

Key Takeaway: "A standby you haven't verified is a second opinion you can't trust. Verification that can fail silently is not verification."

What clears the Staff bar:

• Uses a third replay to arbitrate truth
• Hunts the nondeterministic construct
• Makes verification freshness a failover precondition

Deep Dive 3: Large Member Onboarding#

Context: A major market maker plans to join, expecting to send 3× the messages of your current largest member and wanting a guaranteed latency SLA. Go-live in 8 weeks.

Questions to Surface First:

• Message profile: order-to-trade ratio, cancel rates, burst shape at open?
• Which symbols? Will they concentrate load on a few partitions?
• What latency are they asking for, and can we offer it without disadvantaging other members?
• Kill switch and risk-limit arrangements?

Typical L5 Approach: Provisions extra gateway capacity, load tests their expected rate, signs the SLA. Misses that a latency SLA for one member is a fairness question.

Staff Approach: Dedicated gateway instances (so their bursts don't queue others) with the same hardware and code path as all gateways; certification environment testing with their replayed traffic; per-session throttles and credit limits agreed in writing; the SLA offered is the same published venue SLA, not a special one.

Principal Approach: Ensures fair-access obligations: any latency-affecting product (dedicated gateways, co-location tiers) is offered on published, non-discriminatory terms. Works with compliance so the onboarding doesn't create a two-tier market, and updates capacity planning to model the new member's open-auction bursts.

Key Takeaway: "In an exchange, a special deal for one member is a fairness problem before it's a capacity problem."

What clears the Staff bar:

• Isolates ingress load without changing sequencing fairness
• Certification by replay
• Recognizes fair-access constraints on SLAs

Deep Dive 4: Post-Mortem — Runaway Algorithm Moves the Market#

Context: A member's algorithm malfunctioned for 4 minutes, sending aggressive buy orders across 140 symbols. Prices moved 5–15% in several names before limit-up/limit-down pauses kicked in. Counterparties demand trades be busted.

Questions to Surface First:

• Which controls fired, in what order, and at what seq numbers?
• Why didn't the member's credit limit stop it sooner?
• Did our price collars work as configured?
• What does the rulebook say about clearly erroneous trades?

Typical L5 Approach: Reviews logs, tightens the member's limits, and busts trades that look erroneous. Reasonable, but ad hoc — busting decisions without clear criteria create new disputes.

Staff Approach: Replays the journal to produce the exact control timeline. Finds the credit limit was set on gross notional per symbol, not aggregated across symbols — 140 symbols × per-symbol limit = far too much. Fixes aggregation, adds a cross-symbol velocity check (notional per second per member), and applies the published clearly-erroneous policy mechanically to decide busts.

Principal Approach: Reviews the layered-control model market-wide: which controls are the member's obligation, which the venue's, and which the market's. Proposes default venue-side limits for all members (opt-out requires justification), mandatory kill-switch drills, and a public post-incident summary to maintain market confidence. Knight Capital's 2012 incident is the reference point for why these layers exist.

Key Takeaway: "Controls that are per-symbol can be defeated by breadth. Measure exposure the way the damage accumulates."

What clears the Staff bar:

• Uses journal replay for an exact timeline
• Finds the aggregation gap
• Applies published policy rather than case-by-case judgment

Deep Dive 5: Multi-Region Expansion — A Second Market#

Context: The company wants to launch trading in a second country with its own regulator, and asks whether to reuse the existing engine from the current data center.

Questions to Surface First:

• Does the new regulator require local matching and data residency?
• Latency: members in the new country are 80ms away from the current site. Is that acceptable?
• Different trading hours, tick regimes, auction rules?
• Shared engine codebase vs separate deployments?

Typical L5 Approach: Extends the existing matching site to serve the new market, adding symbols to existing partitions. Simple, but 80ms latency disadvantages local members and likely fails residency rules.

Staff Approach: Separate deployment in the new country — own sequencer, engines, market data, DR site — running the same engine codebase parameterized by market rules as sequenced reference data. Journals stay in-country. Shared tooling and release process.

Principal Approach: Defines the platform/market split: one engine platform team, multiple market operations teams, per-market rulebooks as configuration. Sets the release policy so a change certified for one market can't ship to another without that market's certification. Evaluates whether cross-market products (e.g., cross-listed securities) need a coordination layer — and decides not to build it until a product demands it.

Key Takeaway: "Exchanges don't go active-active across regions; they replicate the platform, not the book."

What clears the Staff bar:

• Rejects cross-region book sharing
• Parameterizes rules as sequenced data
• Separates platform and market ownership

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Explain why a sequencer plus single-threaded matching is the correct architecture, with capacity numbers
• List the constructs forbidden inside a deterministic engine and why
• Describe durability via replicated memory before processing, and bound the residual risk
• Walk through failover by replay, state-hash verification, and the deterministic-bug trap
• Name concrete pre-trade controls with microsecond budgets and a kill switch design
• Design fair market data dissemination with multicast, A/B feeds, retransmit and snapshots
• Treat halt-and-reopen as a designed, rehearsed procedure
• Map engineering ownership to market operations and compliance

The Bar for This Question#

Mid-level (L4): Designs an order service with a database, matches orders with a simple loop, pushes updates via WebSocket. Doesn't address ordering between concurrent orders or recovery.

Senior (L5): Knows price-time priority and a reasonable order book structure, uses a queue to serialize orders, persists trades, adds replicas. The design would work at modest scale. Gaps: parallelizes matching or orders by timestamp, recovers from a database rather than by replay, lacks a failure policy beyond restart, and treats market data as ordinary push notifications.

Staff+ (L6): Builds everything on a sequenced, replicated input log; defends single-threaded matching with numbers; recovers and replicates by deterministic replay with state verification; recognizes that a deterministic bug kills the standby; designs inline risk with µs budgets and kill switches; publishes fair, sequenced market data; and treats halting as a correct outcome. Ownership spans engine, market ops, and compliance. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 The Fastest Exchange Architecture Is the Least Distributed One#

The Staff position: Distribute around the core (gateways, market data, analytics), never inside it.

Why this matters in interviews: It's the clearest test of whether you apply distributed patterns by reflex or by requirement.

10.2 Halting the Market Is a Feature#

The Staff position: Design halt, cancel-only and auction-reopen states as first-class, rehearsed operations.

Why this matters in interviews: "Keep trading at all costs" is web-scale availability thinking misapplied.

10.3 Hot Standbys Give You Less Protection Than You Think#

The Staff position: Pair standbys with replay-in-CI, canary symbols, feature flags per order type, and verification freshness as a failover precondition.

Why this matters in interviews: Naming the deterministic-bug trap is a strong senior-to-staff differentiator.

10.4 Fairness Is the Product; Latency Is Just One Way to Sell It#

The Staff position: Every latency optimization must be evaluated for who it advantages.

Why this matters in interviews: "Faster for whom?" is a Staff question; "how do we make it faster?" is not.

10.5 Kafka Is Not Your Sequencer (For This Latency Class)#

The Staff position: A single-partition log is the right shape; a purpose-built, kernel-bypass sequencer is the right implementation for co-located trading. For a retail or crypto venue with ms budgets, a log like Kafka can be a legitimate sequencer.

Why this matters in interviews: Shows you calibrate technology to latency class rather than rejecting it wholesale.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

At Staff level, the challenge is building a deterministic, fair, recoverable engine. At Principal level, the exchange is a regulated institution whose software is its rulebook. The engine's behavior is what the venue has promised regulators and members; every change is effectively a rule change. The Principal problem is governance of that promise: who can change matching behavior, how changes are proven identical-or-intended, how failure posture is decided and rehearsed across engineering, operations and compliance, and how the business's revenue incentives (data, co-location) are kept consistent with fair-access obligations.

The Org-Level Fault Line#

Engine velocity vs engine sanctity. Product teams want new order types, new auction mechanics and new asset classes quickly. Every change to the engine is a change to regulated behavior and a risk to determinism.

🧭 Principal Move: "The engine team owns a small core and a versioned rules interface. New order types ship as configured behaviors behind flags, certified by full-journal replay and member certification. Nobody else commits to the core."

Cost Model#

Assumptions: fully loaded engineer ~$300K/year; co-location and low-latency network as dominant infrastructure; order-of-magnitude figures.

The dominant costs shift from people (small) to the fairness-and-resilience envelope — co-location, redundant networks, DR, retention (large). Compute for matching itself is a rounding error at every scale.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Matching Engine Determinism and Change Standard (v1)

Scope: All code and configuration executed inside the matching core, and all inputs to it.

MUST:

1. Every input to the engine MUST be a sequenced message, including time, reference data, halts and configuration.
2. Engine code MUST NOT read wall clocks, generate randomness, use floating-point prices, or iterate unordered collections.
3. Every release MUST replay the previous 5 trading days' journals and produce byte-identical outputs, except for explicitly declared behavior changes.
4. New order types and matching behaviors MUST ship disabled and be enabled per symbol via sequenced configuration.
5. Failover MUST be blocked unless the standby's state hash was verified within the last 60 seconds.
6. Halt, cancel-only, and auction-reopen procedures MUST be rehearsed at least monthly in the test environment.

SHOULD: keep engine core under a fixed size budget; run member certification for behavior changes; publish latency percentiles per gateway.

Exceptions: Approved jointly by the engine owner and compliance; documented in the rule filing if behavior-affecting.

Success metrics: zero system-caused busted trades; 100% releases passing replay gate; failover drill success ≥ 99%; median time from order-type request to production.

What I'd Tell the VP#

Our exchange's core is deliberately simple: every order is put in a single official sequence and processed one at a time, which is what lets us prove to regulators and members that everyone was treated fairly. The risks that matter aren't raw speed — they're a software change that alters matching behavior, a backup that silently disagrees with the primary, or a member's runaway algorithm. I'm proposing three controls: every release is tested by replaying real trading days, failover is blocked unless the backup is verified, and halts are practiced monthly. The cost is modest — a few engineers and slower delivery of new order types — against the cost of a single day of busted trades, which is measured in reputation and regulatory action.

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• A perfect engine design with no answer for how the fifteenth order type ships safely
• Failover and halt design without naming who in the organization decides and how it's rehearsed
• Market data design that ignores the business incentive and regulatory fair-access tension

Appendices

Appendix A: Matching Mechanics in Depth#

A.1 Order Book Structure#

A.2 Matching Loop#

on_new_limit(order):                      # called in sequence order only
    book = books[order.symbol]
    opp = book.asks if order.side == BUY else book.bids
    while order.remaining > 0 and opp.best_level() crosses order.price:
        level = opp.best_level()
        resting = level.head()
        if resting.member == order.member and stp_enabled:
            apply_self_trade_prevention(order, resting); continue
        qty = min(order.remaining, resting.remaining)
        emit Executed(seq, resting, qty, level.price)     # passive order's price
        emit Executed(seq, order, qty, level.price)
        emit Trade(seq, level.price, qty)
        order.remaining -= qty; resting.remaining -= qty
        if resting.remaining == 0: level.pop_head()
        if level.empty(): opp.remove_level(level)
    if order.remaining > 0 and order.tif == DAY:
        book.side(order.side).level(order.price).append(order)
        emit BookAdd(seq, order)
    elif order.remaining > 0:                              # IOC remainder
        emit Canceled(seq, order, reason=IOC)

A.3 Cancel-Replace and Priority#

Appendix B: Identity and Data Model#

Appendix C: Sequencing and Replication Mechanisms — Quick Comparison#

C.1 Recovery by Replay#

recover(partition):
    snap = latest_snapshot(partition)          # book state at seq S, hash H
    engine.load(snap); assert engine.hash() == H
    for msg in journal.read_from(S + 1):
        engine.apply(msg)                      # outputs suppressed (already published)
    compare engine.hash() with primary / checkpoint hashes
    resume live at next seq

Appendix D: Member Protocol and Client Behavior#

Appendix E: Observability#

E.1 Core Metrics#

order_ack_latency_us{p50,p99,p999,gateway,member}
stage.queue_delay_us{stage}                 # decode, risk, sequencer, engine, encode
engine.core_utilization{partition}
engine.last_processed_seq{partition}
engine.state_hash_mismatch{partition}
engine.state_hash_compare_age_s{partition}
journal.disk_lag_msgs
sequencer.epoch_changes
md.publish_lag_us{partition}
md.gap_rate{feed}
md.retransmit_requests_per_sec
member.msg_rate / member.exposure_utilization / member.notional_velocity
gateway.p99_skew_us

E.2 Critical Alerts#

E.3 Control Plane vs Data Plane#

Market-ops actions (halt, kill switch, auction triggers, reference data) are control-plane decisions but data-plane messages: they enter through the sequencer so they're ordered, replayable and auditable alongside orders.

Appendix F: Scale Evolution#

F.1 What Works at Each Scale#

F.2 What You Don't Build on Day One#

• Active-active matching across sites
• FPGA-accelerated risk (until kernel-bypass software is measured as the bottleneck)
• Intraday partition rebalancing
• Cross-market coordination
• Custom surveillance ML — start with rule-based alerts on the journal stream

Appendix G: Multi-Member Fairness and Cost#