Technologies referenced in this case study: Redis · Apache Kafka · Cassandra · PostgreSQL · Flink & Stream Processing · ZooKeeper & etcd

Related case studies: Maps & Geospatial · Payment Processing · Real-Time Updates · Stream Processing · Proximity Matching · Notification System

How to Use This Case Study#

Organized for interview use first, reference second. Read once end to end, then drill the sections you're weakest on.

Naive design (nearest-driver query, DB row per trip, synchronous payment):
t=0:       Midnight. Trip requests in one city go 40/s → 600/s in 90 seconds.
t=+10s:    Every request runs "SELECT nearest drivers" against the location table,
           which is also taking 60K GPS writes/s. Lock contention on hot rows.
t=+30s:    Two requests are offered the same driver. Both riders see "Driver on the way."
t=+2min:   Card authorizations to the payment processor time out at 3s. Trip creation
           blocks on payment. Request p99 goes from 400ms to 12s. Riders mash retry.
t=+5min:   Duplicate trips, stranded riders, drivers with two pickups. Support queue x40.

Staff design (in-memory location per city cell, leased offers, async payment saga):
t=0:       Same spike.
t=+10s:    GPS writes land in an in-memory, city-sharded index; durable history is async.
t=+30s:    A driver can hold exactly one offer lease (TTL 15s). The second request
           sees the lease and moves to the next candidate.
t=+2min:   Payment pre-auth runs asynchronously with a risk-based fallback; the trip
           proceeds. Surge rises in hot hexes within ~1–2 min and pulls drivers in.
t=+5min:   Completion rate 92% vs 97% baseline. Degraded, not broken. A metric blip.

Executive Summary

If you only read one section, read this. Everything else in the case study elaborates on the contrast and the positions below.

What This Interview Actually Tests#

Uber is not a "find the nearest driver" question. A geo index answers that in a millisecond.

It is a real-time control loop over mobile supply, wrapped around a durable state machine that moves money — and the design is about keeping the high-rate, lossy parts (location) from contaminating the low-rate, exact parts (assignment, trip state, payment). It tests:

• Whether you see that location ingestion (~250K writes/s at 1M online drivers) dwarfs trip requests (~1K/s at peak) by ~250×, and design each path for its own consistency needs
• Whether you prevent double-assignment of a driver structurally, not with a retry
• Whether you model the trip as an explicit state machine with a single owner, idempotent transitions and a payment saga
• Whether you treat surge as a marketplace control system with feedback, guardrails and regulatory owners — not a multiplier column

The key insight: Location is disposable — the next ping arrives in 4 seconds, so losing one costs nothing. A trip transition is not — losing "driver accepted" or "trip completed" strands a person or loses money. Staff engineers give these two data classes completely different storage, replication and failure behavior.

The L5 vs L6 vs L7 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll design on-demand ride hailing, because it has the tightest latency, the sharpest contention on a single driver, and the most direct money path. Then I'll show what changes for delivery — mainly that the right time to dispatch isn't 'now', it's 'when the food will be ready minus the courier's travel time'. Scheduled logistics is a batch optimization problem and I'd scope it out."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: these are publicly documented, and citing them shows you know how the real systems evolved.

Uber — H3 and Ringpop-Era Dispatch#

Uber open-sourced H3, a hexagonal hierarchical spatial index, and has described using it for pricing and marketplace analysis: hexagons have uniform neighbor distances, which makes supply/demand aggregation and smoothing across neighbors clean. Its earlier dispatch system (publicly described around 2015–2016) sharded state across nodes using Ringpop, a consistent-hashing library with SWIM gossip membership, so each node owned a slice of the geo space.

Staff insight: The shard key for dispatch is geography because the workload is geographic — a request only ever needs drivers within a few kilometers. Say "city/cell is the unit of sharding and of failure" and you've made the most important architecture decision in one sentence.

Uber — Fulfillment Platform Re-architecture on Spanner#

Uber's engineering blog described rebuilding its fulfillment platform — the system that models trips, orders and the entities that fulfill them — onto Google Cloud Spanner to get strongly consistent, transactional updates across entities while scaling horizontally, replacing an earlier design that relied on in-memory state with application-level consistency.

Staff insight: At some scale, "eventually consistent trip state + reconciliation" costs more in engineering and incident time than paying for a strongly consistent store. The trip/order state machine is the one place a transactional store is worth its price.

DoorDash — Dispatch Timed to Food Readiness#

DoorDash has written publicly about its dispatch system (known internally as DeepRed), which uses ML predictions for food prep time and travel time and solves assignment as an optimization problem, including deciding when to dispatch and whether to batch multiple orders onto one courier.

Staff insight: Delivery flips the objective. For rides, dispatching instantly is almost always right. For food, dispatching too early wastes courier time at the restaurant and dispatching too late gives cold food. Mentioning this shows you've understood intents, not just architectures.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: three planes with three different consistency contracts. The location plane is in-memory and lossy on purpose — a lost ping is replaced in 4 s. The dispatch plane is per-city and uses short leases so a driver holds one offer at a time. The trip plane is the only durable, strongly consistent store; every transition writes to it and emits an event through the outbox. Payment and surge are consumers of events, never synchronous dependencies of the rider's request.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: candidates spend 15 minutes on geohash vs quadtree for "nearby drivers" and never reach the trip state machine or payments. The geo index is two sentences. The phases below keep it that way.

Phase 1: Requirements & Framing (2–3 min)#

Functional core in one breath:

"Riders request a ride from A to B, see a price, get matched to a nearby driver, track the driver, take the trip, and pay. Drivers go online, stream location, receive offers, accept, and get paid."

Then the non-functionals, with numbers:

"Let me size the two flows separately. Assume ~1M drivers online at peak, pinging every 4 seconds — that's ~250K location writes/s. Trips are ~25M/day, ~1K requests/s at peak, with New Year's-style spikes of 5–10× in a single city. So location is a ~250:1 firehose relative to trips. Location is disposable — the next ping replaces it. Trip transitions are not — they're money and safety."

Commit to the intent and constraints:

"I'll design on-demand rides. Targets: request-to-assignment p50 under ~15 s, a driver is never assigned to two trips, every trip transition is durable, payment never blocks the ride. I'll come back to what changes for food delivery."

🎯 Staff Move: Naming the 250:1 ratio in the first two minutes tells the interviewer you won't put GPS pings in the trip database — and buys you permission to skip the naive design entirely.

Phase 2: Core Entities & API (1–2 min)#

• DriverLocation: driver_id, lat/lng, heading, speed, ts, h3_cell — latest only, in memory
• DriverState: driver_id, status ∈ {offline, available, offered, en_route, on_trip}, current_trip_id, vehicle_class
• Trip: trip_id, rider_id, driver_id?, state, version, pickup, dropoff, quote_id, fare, timestamps per transition
• Quote: quote_id, price, surge_multiplier, expires_at (~2–5 min)
• PaymentIntent: trip_id, auth_id, amount_authorized, amount_captured, state

POST /v1/quotes        {pickup, dropoff, product}        → {quote_id, price, surge, expires_at}
POST /v1/trips         {quote_id, idempotency_key}       → {trip_id, state: MATCHING}
GET  /v1/trips/{id}                                      → {state, driver, eta, version}
POST /v1/trips/{id}/cancel {version}                      → {state, cancel_fee?}

Driver socket (bidirectional, persistent):
  ← location {lat, lng, heading, speed, ts}               every ~4 s
  → offer {trip_id, pickup, eta, expires_in: 15}
  ← offer_response {trip_id, accept|decline}
  ← trip_action {trip_id, action: arrived|start|complete, version}

🎯 Staff Move: "Every trip mutation carries a version, and creation carries an idempotency key. Mobile clients retry constantly — a rider double-tapping 'Request' in a tunnel must create one trip, and a driver tapping 'Complete' twice must complete it once."

Phase 3: High-Level Architecture (≤ 5 min)#

Three sentences:

"Location ingest writes latest positions to an in-memory index sharded by city — no database on that path. A trip request creates a durable trip in MATCHING and asks the city's dispatcher for a driver; dispatch pulls H3-ring candidates, ranks by ETA, and offers under a 15-second lease. Every trip transition commits to the trip store with an outbox row, and payment, surge, receipts and analytics consume those events asynchronously."

Phase 4: Transition to Depth#

"The nearby search is an H3 k-ring lookup — I don't think that's where the risk is. The three places I'd want to go deep are: preventing double-assignment under a request spike, the trip state machine and how it hands off to payments, and surge as a control loop. I'd suggest starting with double-assignment since it's the failure riders see first."

Phase 5: Deep Dives (25–30 min)#

Double-assignment, as you'd say it:

"Two requests 200 ms apart in the same block both see driver D as the best candidate. Dispatch for each does SET offer:D trip_1 NX PX 15000 on the city's lease store; only one succeeds. The loser takes its next candidate. When D accepts, the Trip Service does a conditional transition: trip_1 from MATCHING to DRIVER_ASSIGNED if the version matches and the lease still names trip_1. If D accepts after the lease expired and trip_1 was reassigned, the accept is rejected and D sees 'offer expired'. At no point can D hold two trips."

Phase 6: Wrap-Up (2–3 min)#

"To summarize: location is an in-memory, lossy, city-sharded firehose; dispatch holds one lease per driver; the trip is a durable state machine with an outbox; payment is a saga that never blocks the ride. What I'd monitor first: double-assignment count (should be zero), assignment latency p50/p99 per city, and payment capture failure rate. What I'd build next: batched matching in the top 20 cities and delivery-specific dispatch timing. What I skipped: route optimization for scheduled logistics and the ETA model itself."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Ride hailing is the canonical "mixed-criticality" system: the same user action touches data that can be lost freely and data that can never be lost. Interviewers use it to see whether you classify data before storing it.

A Senior design that puts all five in one database is overpaying for location and underprotecting payments at the same time.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"If we lose this write, what happens to a person?"

• Lose a GPS ping → the map dot jumps 4 s later. Nothing happens. Keep it in memory.
• Lose an offer → driver misses one request; it's re-offered. Short TTL, no durability.
• Lose "driver accepted" → rider waits for a driver who thinks they're assigned, or two drivers show up. Durable, conditional.
• Lose "trip completed" → rider isn't charged, driver isn't paid, driver can't take the next trip. Durable, idempotent, outbox.

🎯 Staff Move: "I sort every write by what happens to a person if it's lost. That sort gives me three storage tiers, and it's the reason GPS pings never touch the trip database."

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Intent 1 — On-demand ride hailing. One rider, one driver, now. The rider's patience is ~1–3 minutes before they cancel; the driver's attention per offer is ~10–15 s. The system optimizes pickup ETA and match rate under tight latency, with strict single assignment. Pricing adjusts in near-real time to pull supply toward demand.

Intent 2 — On-demand delivery. Three parties: eater, merchant, courier. The critical constraint is food readiness: dispatching a courier the moment an order is placed wastes courier time at the restaurant (idle courier cost) and dispatching too late delivers cold food. Couriers can carry multiple orders (batching), so assignment is a routing problem with pickup/dropoff precedence. ETA accuracy is the product.

Intent 3 — Scheduled logistics. Grocery windows, freight, school buses. Hours of planning horizon, cost-per-stop objective, vehicle routing optimization solved in batch. Real-time tracking matters; real-time matching doesn't.

2.2 When NOT to Use This Architecture#

🎯 Staff Move: "Under ~10K concurrent drivers I'd run this as one service on one Postgres with PostGIS, and pings every 5–10 s. The three-plane architecture is what you grow into when the ping rate stops fitting in a database — I'd name that trigger rather than build for it on day one."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Fault Lines#

3.1 Fault Line 1: Location Freshness vs Ingestion Cost#

The Staff default: latest location in memory, owned by the city/cell shard that dispatches for that area; pings routed by h3_cell → shard through the connection gateway; every ping also appended to Kafka for history, ETA-model training, fraud and fare disputes. Losing a shard's memory costs one ping interval — the ping stream is the replication mechanism.

Adaptive intervals cut ingestion roughly in half: a typical fleet spends ~40–50% of online time idle or parked, where 15–30 s pings are enough.

When to deviate: under ~50K online drivers, Redis GEO per region is simpler and plenty fast (~0.1–1 ms per GEOSEARCH). Move state into the dispatcher only when network hops and Redis hot shards show up in dispatch p99.

🎯 Staff Move: "I don't replicate driver location. The driver's phone replicates it for me every 4 seconds. If a shard dies, its index is rebuilt by the next round of pings — so durability here would be paying for something the protocol already gives us."

3.2 Fault Line 2: Greedy vs Batched Dispatch#

Greedy total pickup ETA: 11 min. Batched: 5 min — at the cost of up to 3 s added to Rider 1's wait.

The Staff default: adaptive. Per H3 res-7 zone, if pending requests × available drivers exceeds a threshold (e.g., ≥ 3 × 3), run a batch every 2–3 s using min-cost assignment on an ETA matrix; otherwise greedy. The ETA matrix is the expensive part: 50 requests × 200 candidates = 10K ETA lookups per batch — served from a precomputed road-graph routing engine with per-segment traffic, ~sub-ms per lookup.

When to deviate: delivery changes the window entirely — the "batch" is minutes and includes when to dispatch. Airport queues are FIFO by regulation or policy, not ETA — a separate dispatcher.

🎯 Staff Move: "Batching is a policy trade: every rider pays ~2–3 s so that the tail rider doesn't pay 7 minutes. I'd only turn it on where density makes the tail real, and I'd measure p90 pickup ETA, not average."

3.3 Fault Line 3: Assignment Consistency — Lease vs Transaction vs Detect-and-Repair#

The invariant: a driver is attached to at most one active trip, and a trip to at most one driver.

The lease is advisory; the trip transition is authoritative:

# Dispatch: claim a driver for an offer
ok = lease_store.set("offer:" + driver_id, trip_id, NX, PX=15000)
if not ok: try next candidate

# Accept: authoritative commit (trip store, per-trip linearizable)
BEGIN
  trip = SELECT ... FROM trips WHERE trip_id = :t FOR UPDATE
  drv  = SELECT ... FROM driver_state WHERE driver_id = :d FOR UPDATE
  if trip.state != 'MATCHING' or drv.status != 'available': ROLLBACK → "offer expired"
  UPDATE trips SET state='DRIVER_ASSIGNED', driver_id=:d, version=version+1
  UPDATE driver_state SET status='en_route', current_trip_id=:t
  INSERT INTO outbox (trip_id, event='driver_assigned', version=...)
COMMIT

Trip and driver state live in the same city-sharded store so this is a single-shard transaction — the reason city sharding matters for correctness, not just scale.

When to deviate: a driver crossing a city boundary mid-shift needs a handoff between shards: the old shard marks the driver migrating, the new shard takes ownership on the next ping, and offers are paused for that driver for one ping interval.

🎯 Staff Move: "The lease stops the obvious race cheaply. But the lease can expire while an accept is in flight, so the accept is a conditional transaction on the trip and driver rows in the same shard. The lease is a hint; the commit is the truth."

3.4 Fault Line 4: Surge — Market Efficiency vs Fairness and Stability#

Surge as a control loop:

every 60–120 s, per H3 res-7 hex h:
  demand   = requests + app_opens_with_quote in last 5 min (h)
  supply   = available drivers + drivers finishing trips within 5 min (h)
  raw      = f(demand / supply)                   # tuned curve, not a ratio
  spatial  = weighted avg of raw over k-ring(h, 1)  # no 2× cliffs between neighbors
  temporal = α · spatial + (1 − α) · previous(h)  # α ≈ 0.3–0.5, damps oscillation
  mult(h)  = clamp(temporal, 1.0, cap(h, time))   # cap from policy; emergency caps
  publish versioned snapshot; quotes embed snapshot version

The Staff point is ownership: the curve f, the cap policy and the emergency override are owned by the Marketplace/Pricing team with Legal sign-off. Uber has publicly committed to capping surge during declared emergencies in some jurisdictions — the kind of external commitment engineering must be able to enforce with a kill switch that works in seconds.

🎯 Staff Move: "Surge is a feedback controller, so I treat it like one: smoothing, damping, bounded output, and a manual override. And the cap table isn't config engineers edit — it's policy with a legal owner."

3.5 Fault Line 5: Trip State Durability vs Availability — and the Payment Handoff#

Payment handoff — the saga:

Rules that make it safe:

• Every processor call carries an idempotency key derived from trip_id + step — retries never double-charge. See Payment Processing.
• Pre-auth failure at request time: for new or risky riders, block the request (fail-closed); for trusted riders, proceed and try again (fail-open) — a risk-policy decision owned by Payments/Risk.
• Capture failure: retry with backoff over hours; try a backup payment method; move trip to ARREARS and block the rider's next request until settled.
• Driver earnings are credited at COMPLETED, not SETTLED. The company absorbs collection risk; the driver is never the victim of a rider's card.

🎯 Staff Move: "The trip and the payment are two state machines linked by events. The trip can finish while the payment is still trying. The only coupling is policy: whether a risky rider can start a trip without a successful pre-auth — and that's a Risk team decision, not a timeout I pick."

4. Failure Modes & Operational Reality#

4.1 Double Assignment Under a Spike#

t=0        Stadium concert ends. 3,000 requests in 4 minutes in 6 hexes.
t=+20s     Lease store (one Redis shard for the city) at 100% CPU; SET NX p99 → 400 ms.
t=+40s     A code path added last month falls back to "offer without lease" on lease
           timeout to "keep matching moving."
t=+60s     148 drivers receive two offers; 61 accept both within the window.
t=+3min    61 riders see a driver who is going somewhere else. Cancellations spike.

• Detection: dispatch.double_offer_total (should be 0), trip.assign_conflict_total (conditional commits rejected), lease.latency_p99.
• Blast radius: one city, event-driven spikes.
• Mitigation: remove the fallback — on lease timeout, skip the candidate, never offer unleased; shard the lease store by H3 res-6; admission control on requests per hex (queue with an honest ETA rather than overload).
• Prevention: invariant test in CI ("no offer without lease"); lease store sized for 10× city peak.
• Owner: Dispatch team.

4.2 Driver Reconnection Storm#

t=0        A cellular carrier has a 3-minute outage across one metro.
t=+3min    Carrier recovers. 40K drivers reconnect within ~10 s.
t=+3min    Connection gateway TLS handshakes spike 50×; ping backlog replays.
t=+4min    Location index for the metro is 3 minutes stale; dispatch offers drivers
           who have since moved 2 km. Pickup ETAs are wrong by 5+ minutes.

• Detection: conn.handshakes_per_s, location.staleness_p99_s{city} > 15 s, dispatch.eta_error_p90.
• Mitigation: clients reconnect with jittered backoff (0–30 s); clients drop queued pings older than 10 s and send only the latest; dispatch excludes drivers with location older than 20 s.
• Prevention: TLS session resumption; per-city connection gateway capacity for 3× reconnect; staleness filter always on.
• Owner: Edge/connection platform (reconnect), Dispatch (staleness filter).

4.3 Surge Oscillation#

• Symptom: a downtown hex flips between 1.0× and 2.8× every 2 minutes; social media screenshots; drivers chase surges that vanish on arrival.
• Root cause: α = 1.0 (no temporal smoothing) after a config change; no neighbor smoothing on a new city launch.
• Detection: surge.multiplier_volatility{hex} (std dev over 15 min), surge.neighbor_delta_max.
• Mitigation: restore smoothing; freeze surge snapshot for the city while investigating.
• Owner: Pricing team; config change process owned with Marketplace.

4.4 Payment Processor Degradation#

t=0        Card processor p99 goes 300 ms → 8 s in one region.
t=+1min    Pre-auth calls time out; if the request path awaits pre-auth, requests stall.
t=+5min    Capture retries pile up; outbox relay lag grows.

• Detection: payment.auth_latency_p99, payment.capture_retry_queue_depth, outbox.relay_lag_s.
• Blast radius: with the saga design — none to trips; collection is delayed. With a synchronous design — every new request in the region.
• Mitigation: trust-tier policy: trusted riders proceed without completed pre-auth; circuit-break the processor and route to a secondary processor where contracted; capture retries with exponential backoff over hours.
• Owner: Payments team; risk policy owned by Risk.

4.5 Trip Store Region Failure Mid-Trip#

• Scenario: the home region for 30 cities loses its trip store primary. ~200K trips are in progress.
• What must not happen: trips vanish; drivers can't complete; riders are double-charged after recovery.
• Mitigation: trip store replicated synchronously across zones within the region (survives a zone); for a region loss, promote the async replica in the paired region (RPO of seconds). Clients cache the current trip state and version locally and re-submit their last action with the same idempotency key after failover. Transitions lost in the RPO window are reconstructed from client-held state and location history (e.g., "completed" inferred from driver app + GPS at dropoff).
• Detection: trip.write_errors{region}, trip.client_resubmits_total.
• Owner: Trip platform + SRE.

4.6 GPS Drift and Fare Disputes#

• Symptom: urban canyons produce 50–100 m jumps; naive distance summation inflates fares by 5–15% on some routes.
• Mitigation: map-matching to road segments before distance calculation; upfront pricing so the fare is mostly fixed at request; disputes resolved from matched route.
• Owner: Maps/Routing team; Fares team.

4.7 Operational Reality Matrix#

🎯 Staff Move: "ETA fails open to a crude estimate, location fails to one-ping staleness, payments fail to arrears — but trip transitions never fail open. That's the one place we'd rather reject a request than guess."

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Geo-index fluency ≠ dispatch design. Knowing H3 resolutions is table stakes.
• "We'll use Kafka" ≠ event-driven correctness. Without an outbox, state and events diverge.
• Optimization vocabulary (Hungarian algorithm) ≠ marketplace judgment. The signal is knowing when batching's latency is worth it.
• Multi-region buzzwords ≠ trip durability. "Active-active" without saying who owns a trip mid-failover is a red flag.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

6.4 Follow-Up Questions to Expect#

1. "Two riders request within 100 ms and the best driver is the same. What happens?"
2. "A driver accepts, then their phone dies. What does the rider see, and when?"
3. "How many writes per second hit your system, and which are durable?"
4. "The card capture fails. Is the driver paid?"
5. "Surge in one hex is 3× and the next hex is 1×. Is that a problem?"
6. "A region dies. What happens to trips in progress?"
7. "How would you change dispatch for food delivery?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design Uber."

Drill 2: Core Mechanic — Driver Accepts, Phone Dies#

Prompt: "The driver taps Accept and their phone dies before the response arrives. What happens?"

Drill 3: Make It Concrete — Location Ingestion Sizing#

Prompt: "Size location ingestion and tell me where every byte goes."

Drill 4: Dependency Down — ETA Service Outage#

Prompt: "The ETA service is returning errors for one city."

Drill 5: Hot Key — Airport Queue and Stadium Exit#

Prompt: "500 drivers wait at an airport lot; 3,000 riders request at once after a concert. What's hot?"

Drill 6: Multi-Tenant — Rides and Delivery on One Platform#

Prompt: "Should rides and delivery share dispatch?"

Drill 7: Build vs Buy — Trip State Store#

Prompt: "Build your own trip store on sharded MySQL, or buy something like Spanner?"

Drill 8: Policy Change Without Outage — New Surge Curve#

Prompt: "Pricing wants a new surge curve launched Friday."

Drill 9: Cost#

Prompt: "Infra costs per trip rose 40%. Where do you look?"

Drill 10: Multi-Region#

Prompt: "Make it survive a region outage."

8. Deep Dive Scenarios#

Deep Dive 1: New Year's Eve — Assignment Latency Collapse#

Context: 00:02 local, one of your top-5 cities. Request-to-assignment p50 jumped from 12 s to 90 s; cancellations are 4× normal. The on-call escalates to you.

Questions to Surface First:

• Is supply exhausted (no available drivers) or is dispatch slow (drivers available but not offered)?
• What are lease store and dispatch shard latencies in the hot zones?
• Is the batch solver timing out?
• Is surge responding, and are drivers moving toward hot hexes?

Typical L5 Approach: Scales the dispatch service horizontally and increases the lease store's memory. Doesn't check whether the solver or the supply is the bottleneck.

Staff Approach: Splits the problem in 2 minutes: drivers_available{zone} vs dispatch.pending_requests{zone}. Finds drivers available but solver batches taking 8 s (candidate set exploded because requests widened search radius). Caps candidates per request to 30 by ETA, shrinks batch window to 1 s in the hot zones, and turns on pickup-zone steering. Surge is at cap; confirms caps are policy, not a bug.

Principal Approach: Asks why the city's top event of the year wasn't in an event calendar with pre-provisioned solver capacity and pre-tuned parameters. Makes "known peak events" a planning artifact owned jointly by Marketplace Ops and Dispatch, reviewed 2 weeks ahead, with a load test at 3× last year's peak.

Key Takeaway: "When a solver's inputs grow, its runtime grows faster. Bound the inputs before you scale the fleet."

What clears the Staff bar:

• Separates supply shortage from dispatch slowness
• Bounds solver inputs
• Turns manual switch into an automatic guardrail

Deep Dive 2: The Silent Failure — Drivers Unpaid for Completed Trips#

Context: Driver support tickets show ~0.3% of completed trips missing from weekly earnings. No alerts fired.

Questions to Surface First:

• Are the trips COMPLETED in the trip store?
• Did the trip_completed events reach the earnings consumer?
• Is there an outbox relay gap or a consumer skipping on error?
• Is it correlated with a region, app version, or payment status?

Typical L5 Approach: Finds a consumer bug that dropped events on deserialization errors and backfills the missing earnings.

Staff Approach: Finds the root: the earnings consumer caught schema errors after a trip event schema change (new optional field with a non-default type), logged them, and committed offsets — silently dropping events. Backfills from the outbox table (the source of truth). Adds a DLQ with an alert on depth > 0, and a daily reconciliation: COMPLETED trips vs earnings lines per driver per day; mismatch > 0 pages.

Principal Approach: Treats the trip event as a versioned public contract. Establishes a schema registry with compatibility enforcement in CI for all 20+ consumers, and a rule that money-moving consumers never skip — they stop and page. Makes the earnings reconciliation report visible to Driver Operations leadership.

Key Takeaway: "A consumer that moves money must never skip an event. It stops, and someone gets paged."

What clears the Staff bar:

• Uses the outbox as a source of truth for recovery
• Adds reconciliation, not just a bug fix
• Classifies consumers by criticality

Deep Dive 3: Large-Customer Onboarding — Enterprise Delivery Partner#

Context: A national grocery chain wants to route 200K orders/day through your delivery platform, with scheduled windows and bulk orders (courier capacity limits matter).

Questions to Surface First:

• What fraction is on-demand vs scheduled windows?
• Vehicle capacity constraints (a 40-item order doesn't fit on a bike)?
• SLA commitments and penalties?
• Integration: API volume, order bursts at window open?

Typical L5 Approach: Adds a bulk-order API and relies on existing on-demand dispatch.

Staff Approach: Recognizes that scheduled windows are Intent 3 — a batch routing problem. Runs a VRP planner per window 60–90 min ahead, then hands planned routes to the real-time layer for execution and exception handling. Adds vehicle capacity as a dispatch constraint. Rate-limits the partner's order API with a per-tenant quota and burst allowance. Negotiates SLAs based on measured percentiles, not averages.

Principal Approach: Evaluates whether this partner justifies a scheduled-logistics platform the company doesn't have yet, prices it (team of ~8 for a year), and decides with the business whether to build, buy a routing vendor, or decline scheduled windows in the contract.

Key Takeaway: "A big customer can quietly change your intent. Recognize when on-demand dispatch is the wrong tool."

What clears the Staff bar:

• Detects the intent shift
• Tenant isolation
• SLA grounded in measurement

Deep Dive 4: Post-Mortem — Riders Double-Charged After Failover#

Context: After a regional failover, ~12K riders were charged twice. Press is asking.

Questions to Surface First:

• Were two captures issued, or one capture plus a new auth converted?
• Were idempotency keys used, and were they scoped per processor account?
• Did the payment saga state replicate with the trip state?

Typical L5 Approach: Refunds duplicates and adds a dedupe check before capture.

Staff Approach: Finds that the saga state was in a region-local store not replicated to the paired region; after failover the saga restarted for trips in COMPLETED and generated new idempotency keys with a random component. Fix: idempotency keys are deterministic (trip_id:capture:v1) and processor-side keys dedupe regardless of region; saga state replicates with the trip store; a pre-capture check reads the processor's charge list for the trip reference.

Principal Approach: Establishes an org-wide rule: any retry-capable money operation uses deterministic idempotency keys derived from business identifiers, audited in code review; failover game days include payments. Coordinates rider communication and refunds with Comms, and reports to the finance controller.

Key Takeaway: "Idempotency keys must be derived from the business event, never generated at retry time."

What clears the Staff bar:

• Pinpoints key generation as root cause
• Replicates money state with trip state
• Adds classification to prevent recurrence

Deep Dive 5: Multi-Region Expansion — Launching Southeast Asia#

Context: Launching in 6 cities across 3 countries with different payment rails (cash common), residency rules, and motorbike taxis.

Questions to Surface First:

• Residency: must trip/location data stay in-country?
• Cash payments: how does the saga change?
• Vehicle types: motorbikes change ETA and pickup behavior.
• Network: 3G fallback, high packet loss?

Typical L5 Approach: Deploys the existing stack in a Singapore region.

Staff Approach: Region cell in-region; per-country residency where required. Cash changes the payment saga: no pre-auth; completion records cash collected; the company's commission becomes a driver debt settled from future card trips or top-ups — a new state machine. Motorbike ETA model and vehicle class. Client: lower ping frequency on poor networks, compact binary payloads.

Principal Approach: Decides build-vs-partner per market (local wallets, maps), and sets a region-launch template. Assesses regulatory posture on surge per country before launch.

Key Takeaway: "A new payment rail isn't a config change — it's a new state machine."

What clears the Staff bar:

• Payment-rail-driven design changes
• Vehicle-specific models
• Regulatory review before launch

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Classify ride-hailing data by loss tolerance and design three planes accordingly
• Size location ingestion with adaptive ping rates
• Prevent double assignment with leases plus a conditional commit
• Draw the trip state machine including cancellations, no-shows and payment settlement
• Design the payment handoff as a saga with deterministic idempotency keys
• Explain surge as a smoothed, capped control loop with a legal owner
• Explain how delivery changes dispatch timing
• At L7: price the planes, draw platform vs product ownership, and plan the 3-year evolution

The Bar for This Question#

Mid-level (L4): Rider/driver/trip tables, nearest-driver query, charge at end. Works for one city of 100 drivers.

Senior (L5): Adds geohash/H3, Redis for location, Kafka, sharding. Knows about idempotency. Keeps payment synchronous, uses a status column, and misses double assignment or solves it with a global lock.

Staff+ (L6): Separates planes by loss tolerance, prevents double assignment structurally, models the trip as a state machine with outbox and payment saga, treats surge as a controller with policy owners, and uses the city as the unit of sharding and failure. Every failure has a metric and an owner. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Don't Replicate Driver Location"#

The Staff position: In-memory latest location; history async for analytics.

Why this matters in interviews: It shows you reason from the protocol, not from habit.

10.2 "Batched Matching Is Overrated Outside Dense Cores"#

The Staff position: Adaptive: batch where density justifies, greedy elsewhere.

Why this matters in interviews: Candidates who say "Hungarian algorithm" confidently rarely say where not to use it.

10.3 "Surge Is a Legal Feature, Not a Pricing Feature"#

The Staff position: Caps, overrides and audit trails are first-class requirements.

Why this matters in interviews: Naming the non-engineering owner is a Staff-level signal.

10.4 "The Trip State Machine Deserves the Most Expensive Database You Have"#

The Staff position: Cheap stores for location; strongly consistent store for trips.

Why this matters in interviews: Spending asymmetrically is the Staff move.

10.5 "Driver Earnings Must Never Depend on Rider Payment Success"#

The Staff position: Credit at completion; company owns collection.

Why this matters in interviews: It shows you name who pays.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer designs a ride-hailing system. A Principal engineer sees a fulfillment company: rides, food, grocery, parcels and freight all assign mobile supply to located demand, track it, price it and pay it. The expensive, slow-to-build assets — the location plane, the road graph and ETA models, the payment and payout rails, the trip/order state machine framework — are shared, and every product team will try to fork them to move faster. The L7 question is how to keep one ETA, one location truth and one money path while letting products compete on dispatch policy and UX — and which decisions (city-as-cell, trip store technology, event contracts) will lock the company in for five years.

The Org-Level Fault Line#

One fulfillment platform vs per-vertical stacks.

🧭 Principal Move: "The platform owns anything where two versions of the truth would embarrass us in front of a customer — ETA, location, money. Verticals own anything that is their competitive difference — dispatch objective, batching, UX. The lease store is global per driver because the driver is a shared resource."

Cost Model#

Assumptions: cloud list prices order-of-magnitude; adaptive pings averaging ~5 s; ~150-byte pings; strongly consistent trip store; loaded engineer ~$250K/yr.

The line that moves the business case: ETA lookups per dispatch. A candidate-set or batch-size change can move infra cost by double-digit percent — ETA request volume should be a budgeted, reviewed quantity.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Fulfillment State & Money-Movement Standard (v1)

Scope: Every service that creates or transitions a trip, order or delivery, or that moves money as a result.

Mandatory requirements:

• Fulfillment entities MUST be modeled as explicit state machines in the shared framework; transitions MUST be versioned and validated server-side.
• State changes MUST emit events via a transactional outbox; direct dual writes to a database and a broker are prohibited.
• Money-moving calls MUST use deterministic idempotency keys derived from (entity_id, step, version).
• Money-moving consumers MUST NOT skip events on error; they halt and page.
• A supply unit (driver/courier) MUST hold at most one active offer across all verticals, enforced by the global lease service.
• Location history SHOULD be downsampled outside active trips and deleted after the retention set by Privacy.

Exceptions: Filed with Fulfillment Platform and Payments; time-boxed to one quarter.

Success metrics: 0 double assignments per quarter; 0 duplicate captures; earnings reconciliation mismatch = 0 for 30 days; 100% of verticals on the shared state framework within 4 quarters.

What I'd Tell the VP#

"Our riders and drivers only notice three things: how fast they're matched, whether the price makes sense, and whether the money is right. Today each of our apps builds its own version of those systems, which is why ETAs sometimes disagree and why a failover double-charged customers last quarter. I'm proposing one fulfillment platform that owns location, ETAs, trip state and money movement, with each product team free to design its own dispatch rules on top. It costs roughly 30 engineers we mostly already have, reorganized. The payoff is fewer money incidents, faster launches of new verticals, and one set of numbers our drivers can trust."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll use an outbox." — Correct, but doesn't make it the org standard or govern the event schema consumed by 20 teams.
• "Surge has caps." — Correct, but doesn't say who owns the caps across jurisdictions or how emergency commitments are enforced.
• "Delivery can reuse dispatch." — Correct, but doesn't address shared supply, cross-vertical priority, or who arbitrates.

Appendices

Appendix A: Mechanics in Depth#

A.1 Candidate Generation#

cells      = h3.k_ring(h3.cell(pickup, res=9), k=ring_for(radius))   # k=5 ≈ 1.5 km
candidates = [d for c in cells for d in index[c]
              if d.status == 'available' and now - d.ts < 20s
              and d.vehicle_class in product.classes]
candidates = top_n_by(haversine, candidates, 100)      # cheap prefilter
etas       = eta_service.matrix(candidates, pickup)     # one batched call
ranked     = sort_by(eta + deadhead_penalty + fairness_adj, candidates)[:30]

A.2 Batched Assignment#

For a zone every 2–3 s: build cost matrix C[r][d] = eta(d, r) + penalties; add dummy columns with a high "wait for next batch" cost; solve min-cost assignment (Hungarian O(n³) is fine for n ≤ ~200; beyond, partition by zone). Requests unassigned roll into the next batch; after ~3 batches, fall back to greedy with a widened radius.

A.3 Surge Smoothing#

Temporal EWMA with α ≈ 0.3–0.5 over 60–120 s cycles; spatial averaging across the k=1 ring; hard clamp by policy cap; emergency override sets cap = 1.0 for a polygon in < 60 s.

Appendix B: Keys and Data Model#

Appendix C: Coordination Mechanisms — Quick Comparison#

Appendix D: API Contract and Client Behavior#

• Trip creation requires idempotency_key; duplicates return the existing trip.
• All trip commands carry version; stale versions return 409 with current state.
• Driver socket: server sends offers with expires_in; client must respond within it; late accepts return offer_expired.
• Reconnect: jittered exponential backoff (base 1 s, cap 30 s); on reconnect send only the latest location; fetch current trip state by version.
• Rider tracking: server pushes driver position every 1–2 s during approach; client interpolates.
• Quotes: valid until expires_at; trip creation with an expired quote returns a fresh quote for confirmation.

Appendix E: Observability#

E.1 Core Metrics#

E.2 Critical Alerts#

E.3 Control Plane vs Data Plane#

Data plane: pings, offers, transitions, captures. Control plane: surge curves and caps, dispatch parameters (batch window, candidate caps), city-to-region mapping, ping policies. Control-plane changes are versioned, canaried by city, and reversible within one cycle.

Appendix F: Scale Evolution#

F.1 What You Don't Build on Day One#

• Batched matching
• In-process location sharding
• Multi-region active failover
• Custom routing engine (use a maps provider)
• Delivery-specific dispatch

What you do build on day one: the explicit trip state machine with outbox, deterministic idempotency keys for money, and driver earnings decoupled from rider payment.

Appendix G: Multi-Tenancy, Fairness and Cost#

• Driver fairness: track trip distribution and earnings per online hour across drivers in a zone; dispatch adds a small fairness adjustment for drivers idle longest.
• Rider fairness: monitor pickup ETA by neighborhood; dispatch radius policies should not systematically exclude lower-density areas.
• Tenant isolation (enterprise delivery partners): per-tenant API quotas and burst allowances; per-tenant SLO dashboards.
• Cost attribution: cost per trip by plane and by vertical, reported monthly; ETA lookups per dispatch budgeted.