Dealing with Contention — Cross-Cutting Pattern

Technologies that implement this pattern: PostgreSQL · Redis · DynamoDB · ZooKeeper & etcd · Apache Kafka · Cassandra

Why This Matters#

Contention is what happens when the business sells something scarce and many people want it in the same second. The last 200 concert seats, the one driver near the airport, the final unit of a sneaker drop, the ledger row for a merchant doing 3,000 payments a minute. Every one of these collapses to the same physical fact: many writers, one piece of state, and an invariant that must hold — never sell 201 seats, never assign the driver twice, never let the balance go negative.

Most candidates treat contention as a locking question: "I'll take a lock on the row." Staff engineers treat it as a data modeling question. A lock does not remove contention; it orders it, and ordering has a throughput ceiling of roughly 1 / critical_section_duration. If the critical section is 5ms, the ceiling is ~200 operations per second on that key, forever, no matter how many servers you add. The durable fixes change the shape of the state so that fewer writers touch the same thing: split the inventory into buckets, turn "decrement the counter" into "claim a pre-minted token", route every writer for a key to one owner, or move the scarce decision out of the database and into an admission queue.

The second reframe: contention is a fairness and user-experience problem as much as a correctness problem. Two designs can both guarantee zero oversell. One makes 95% of buyers stare at a spinner for 40 seconds and then fail; the other tells them in 200ms that they are #18,402 in line. The invariant is the floor. What the losers experience is the design.

If you can walk an interviewer from "what is the invariant and how hot is the key" to "which coordination primitive matches that heat" to "what the losers see and who gets paged when the invariant breaks", you are answering at Staff level.

The 60-Second Version#

• Measure the heat before picking a primitive. Contention is writes/sec on one key × critical section duration. Below ~1 (e.g., 50 writes/sec × 5ms = 0.25) almost anything works. Above ~1, queueing dominates and latency grows without bound.
• Atomic conditional writes beat locks for single-row invariants. UPDATE inventory SET qty = qty - 1 WHERE sku = $1 AND qty > 0 holds the row lock for one statement (~0.2–1ms), not a whole transaction with a network round trip in the middle (~5–50ms). That is a 10–50× higher ceiling for free.
• Optimistic concurrency works below ~5–10% conflict rate. Above ~20–30%, retries amplify load: with k contenders per window, expected attempts grow roughly linearly in k, and a retry storm can take a 2K-QPS endpoint to 20K QPS of wasted work.
• Hot single keys need structural fixes, not faster locks. Split a counter of 10,000 seats into 32 buckets of ~312 each and you get ~32× the write ceiling; the price is a slightly more complex "sold out" check and stranded units in empty-adjacent buckets.
• Distributed locks need fencing tokens or they are advisory. A lease-based lock (Redis, etcd, ZooKeeper) can expire while the holder is paused (GC, VM stall, 10–30s is not rare). Without a monotonically increasing token checked by the resource, two holders will write.
• Holds are inventory too. A 10-minute checkout hold at 50K concurrent carts locks up 50K units. Hold TTL, release-on-abandon, and the oversell/undersell ratio are product decisions with revenue attached.

The Problem#

A ticketing platform puts 20,000 seats on sale at 10:00:00. At 10:00:01 it has 600,000 people clicking "buy". A naive design reads available seats, picks one, and writes the purchase — and at that concurrency two buyers read the same free seat before either writes, so seat 14C is sold twice. The "fix" of wrapping it in SELECT ... FOR UPDATE serializes every buyer behind one lock on the inventory row; at ~8ms per transaction that is ~125 purchases per second, the queue behind the lock grows by ~600,000 in the first second, connection pools exhaust, and the database stops serving everything else — including the payment confirmations for people who already won. The same shape shows up in a payment ledger with one hot merchant balance, a ride dispatcher with one driver and five matching requests, a coupon with a 1,000-redemption limit, and a rate-limit counter for a tenant doing 50K requests per second. The job is to keep the invariant, keep throughput, and give every contender a fast, honest answer.

Case Studies That Use This Pattern#

• Flash Sales & Ticketing — The extreme case: 100:1 demand over supply in one second; admission queues, bucketed inventory, and what the losers see
• Reservation Systems — Holds with TTLs, double-booking prevention, and the hold-vs-commit split that makes contention survivable
• Distributed Lock Service — Leases, fencing tokens, and why "acquire lock" is never the whole answer
• Payment Processing — Hot merchant balances and ledger rows; append-only entries instead of in-place balance updates
• Ride Hailing & Delivery — One driver, many requests: single-owner matching per geo cell instead of row locks
• Stock Exchange — The purest single-writer design: one sequencer per instrument, zero locks
• Rate Limiting — A hot counter per tenant; local-first counting to avoid one Redis key absorbing every request
• Idempotency & Exactly-Once — Retries under contention create duplicates; dedupe keys are part of the contention design

The Four Intents#

"Handle concurrent purchases" hides at least four goals that lead to incompatible designs. Name them and commit.

🎯 Staff Move: "I'll treat the seat inventory as intent one — zero oversell, product and legal sign off on that — and the 'people waiting' counter as intent two, where ±2% is fine. Those get different mechanisms. And because demand is 30× supply, I'm putting an admission queue in front so most of the contention never reaches the database."

The Core Tradeoff#

Staff Default Position#

Push the invariant into one atomic operation on one owner, and keep the losers from ever reaching that owner.

Start by asking whether the invariant fits a single row or item. If it does, use an atomic conditional write — no read-then-write, no transaction spanning a network call. If one key is hotter than one row can absorb (~1–5K conditional updates/sec on Postgres, ~50–100K on a single Redis key), split it into buckets or route it to a single-writer owner. When demand exceeds supply by 10× or more, put an admission layer in front so that the database sees roughly supply × small multiple attempts, not demand. Use pessimistic locks only for genuinely multi-row invariants with low heat, and use distributed locks only with fencing tokens validated by the resource. Every hold has a TTL, an owner, and a metric for leaked inventory.

When to Deviate#

• Multi-entity invariants that cannot be collapsed — Transferring between two accounts, booking a flight + hotel. Use a short transaction with locks taken in a canonical order (sorted by ID) to avoid deadlocks, or a saga with reservations if the entities live in different services. Accept the lower ceiling and keep the critical section free of network calls.
• Low heat — Under ~50 writes/sec per key, SELECT ... FOR UPDATE in a 2ms transaction is fine and is the easiest thing for the next engineer to read. Don't build a bucketed inventory system for a B2B app with 40 orders a day.
• Approximate is acceptable — Like counts, view counts, soft quotas. Use local aggregation and periodic flush; don't coordinate at all. Product signs off on the error bound.
• Correctness must survive a region loss — If the invariant must hold across regions with no single home, you need consensus-backed storage (Spanner, CockroachDB, etcd) and you pay 50–150ms of cross-region latency per commit. Usually better: give each scarce resource a home region.

The L5 → L6 → L7 Contrast#

The Five Fault Lines#

Fault Line 1: Pessimistic vs Optimistic#

Pessimistic locking makes every contender wait in line; optimistic concurrency lets everyone proceed and rejects the losers at commit. The deciding number is the conflict rate. At 2% conflicts, OCC wastes 2% of work and never blocks. At 40%, each success costs ~2.5 attempts and the retries themselves raise the conflict rate — a positive feedback loop. Pessimistic locks have the opposite profile: zero waste, but throughput is pinned at 1 / hold_time and every waiter holds a connection. Who pays: under OCC, the clients who keep losing (and the backend absorbing retries); under locks, every contender's p99 and the shared connection pool. Staff default: atomic conditional writes first (they are "optimistic" with a retry-free loser path: the update simply affects 0 rows), OCC with version columns for read-modify-write on low-heat entities, locks for short multi-row invariants. Deviate when: conflict rate is measured above ~20% — switch to a single-writer or a queue rather than tuning retry backoff.

Fault Line 2: One Hot Key vs Split State#

One row with qty = 10000 is easy to read and easy to reason about, and its write ceiling is one row's. Split it into 32 rows of ~312 and the ceiling rises ~32×, but now "how many are left?" is a sum, "sold out" means all 32 buckets are zero, and the last ~1% of units may sit in a bucket nobody is routed to. Who pays: product, at the end of the sale (the last 50 units take longer to sell; some buyers see "sold out" while 3 units remain); the read path (fan-in of N rows or a periodically refreshed total). Staff default: split when a key exceeds ~50% of its measured single-row ceiling at peak; route buyers to a random bucket and probe 2–3 more on empty before declaring sold out; rebalance stragglers into one bucket when total remaining < N × 5. Deviate when: the invariant spans the whole quantity in a way that can't be partitioned (e.g., a strict global ordering among buyers) — then single-writer.

Fault Line 3: Coordinate in the Store vs Coordinate in Front#

When 600K people want 20K seats, letting all 600K attempts reach the database is the mistake — even if each attempt is a cheap atomic update, 580K of them are doomed and they crowd out the 20K that matter (plus every other workload on that cluster). An admission layer — a virtual waiting room, a token bucket per event, a pre-minted pool of "purchase tokens" in Redis — lets only ~supply × 1.2–2 contenders through. Who pays: the admission layer adds a system that can fail, a fairness policy someone must defend, and wait time for users; coordinating in the store makes the database the bottleneck for everyone. Staff default: if demand/supply > ~10× or peak attempts exceed ~20% of the store's write capacity, put admission in front. Deviate when: demand is steady and only moderately above supply (e.g., a restaurant booking system) — atomic writes in the store are enough.

Fault Line 4: Lease Safety vs Liveness#

A lease-based lock grants ownership for T seconds. Long T (60s) survives a 20s GC pause but means a crashed holder blocks the resource for up to a minute. Short T (5s) recovers fast and guarantees that a paused holder will, sooner or later, wake up believing it still owns the lock. No choice of T is safe on its own; fencing tokens make it safe: every grant carries a monotonically increasing number, and the protected resource rejects writes with a token lower than the highest it has seen. Who pays: without fencing, data correctness; with long TTLs, availability of the protected resource. Staff default: etcd/ZooKeeper-backed leases (consensus, monotonic revision numbers) with 10–15s TTL, renewal at TTL/3, and the token checked by the storage write (WHERE fence < $token). Deviate when: the lock only prevents duplicated effort (two workers computing the same report) rather than duplicated effects — then an unfenced Redis SET NX PX is fine and cheap.

Fault Line 5: Hold Generously vs Commit Fast#

A reservation hold lets a buyer pick a seat and enter payment details without losing it. A 15-minute hold is buyer-friendly; at 50K concurrent checkouts it removes 50K units from sale, and if 40% abandon, 20K units sit dead until expiry — during the exact minutes demand peaks. Who pays: short holds — buyers who lose the seat while typing a card number (and support, who hears about it); long holds — the business (unsold inventory, bots holding seats to resell). Staff default: 5–8 minute hold for high-demand events, 15–20 for low-demand; one active hold per user per event; releases on explicit abandon, on payment failure, and on TTL via a sweeper that runs every 10–30s. Deviate when: payment is instant (stored card, one-click) — hold for 60–90s or skip the hold and commit atomically.

The hold lifecycle is where most contention bugs live: the transitions back to Available are written by sweepers, payment callbacks and admin tools, and every one of them must be the same atomic, idempotent operation.

Common Interview Mistakes#

Quick Reference#

Staff Sentence Templates#

"The invariant here is [never sell more than N / never double-assign]. The hottest key sees [X] writes/sec at peak, and my critical section is [Y ms], so a lock would cap me at [1000/Y] per second. That's [above / below] peak, so I'll use [conditional update / buckets / single writer]."

"Demand is about [D]× supply. Letting every attempt hit the database means [D−1] out of [D] writes are doomed and they crowd out the winners. I'll admit roughly [1.5]× supply through a queue and tell everyone else their position within [200ms]."

"This lock protects [effects / effort]. Because it protects effects, the lease alone isn't safe — the storage write checks a fencing token, so a holder that was paused for [N seconds] gets rejected instead of corrupting [resource]."

"Hold TTL is a product decision priced in stranded inventory: at [C] concurrent checkouts and [A]% abandonment, a [T]-minute hold keeps [C × A] units off sale at peak. [Product owner] signed off on [T]."

Implementation Deep Dive#

1. Atomic Conditional Decrement — PostgreSQL#

The single most valuable move in contention design: turn read-check-write into one statement. The row lock is held for one statement, not across a round trip to the application.

-- Claim one unit; rows affected = 1 means you won, 0 means sold out
UPDATE inventory
   SET available = available - 1,
       version   = version + 1
 WHERE event_id = $1
   AND bucket   = $2
   AND available > 0
RETURNING available;

-- Same transaction: record who won, with an idempotency key
INSERT INTO holds (hold_id, event_id, bucket, user_id, expires_at)
VALUES ($hold_id, $1, $2, $user, now() + interval '6 minutes')
ON CONFLICT (hold_id) DO NOTHING;

-- Belt and braces: the schema enforces the invariant even if a new code path forgets
ALTER TABLE inventory ADD CONSTRAINT available_nonneg CHECK (available >= 0);

Numbers: a single-statement update on an indexed row with synchronous commit costs ~0.5–2ms of lock hold on NVMe; a single hot row tops out somewhere around 1–5K updates/sec depending on commit latency and group commit. The CHECK constraint is the invariant's last line of defense — it turns an oversell bug into an error log instead of a refund.

🎯 Staff Insight: Notice what's missing: no SELECT, no application-side check, no lock spanning a network call. The losers get 0 rows immediately and can be told "sold out" in one round trip. The fastest contention design is the one where losing is cheap.

2. Bucketed Inventory — Redis + Lua#

When one key exceeds what a single row or single Redis key can absorb, split it. Redis executes Lua scripts atomically, so each bucket claim is one indivisible operation.

# Setup: 20,000 seats split across 32 buckets on different hash slots
for i in 0..31:
    redis.SET("ev:{" + event + ":" + i + "}:avail", 625)

# claim.lua — atomic per bucket
local n = tonumber(redis.call('GET', KEYS[1]) or '0')
if n <= 0 then return -1 end
redis.call('DECR', KEYS[1])
redis.call('SET', KEYS[2], ARGV[1], 'PX', ARGV[2], 'NX')   -- hold:{id} with TTL
return n - 1

function claim(event, user):
    tried = 0
    for b in shuffle(0..31).take(4):                # probe at most 4 buckets
        r = redis.EVALSHA(claim_sha,
              keys=["ev:{" + event + ":" + b + "}:avail", "hold:" + uuid()],
              args=[user, 360000])
        if r >= 0: return Hold(event, b)
        tried += 1
    if total_remaining(event) == 0: return SOLD_OUT  # sum of 32 GETs, cached 250ms
    return TRY_AGAIN                                  # stragglers being rebalanced

Why it matters: a single Redis key sustains roughly 50–100K simple ops/sec on one core; spreading across 32 slots on an 8-shard cluster raises the ceiling ~8×, and more importantly no single shard becomes the hot spot for the event. The Redis count is the fast gate; the durable record (Postgres) is written asynchronously from the hold stream and reconciled — redis_sold - db_sold is a metric that must stay at zero after the sale.

3. Optimistic Concurrency — DynamoDB Conditional Writes#

For read-modify-write on entities with low conflict (a profile, a cart, a booking with several fields), version-based OCC avoids locks entirely.

item = ddb.GetItem(Key={pk: "booking#" + id}, ConsistentRead=true)
new  = apply_change(item)            # arbitrary business logic, no lock held
try:
    ddb.PutItem(
        Item = new with version = item.version + 1,
        ConditionExpression = "version = :v",
        ExpressionAttributeValues = {":v": item.version})
except ConditionalCheckFailed:
    metrics.incr("occ.conflict", tags=["entity:booking"])
    if attempt < 3:
        sleep(random(10ms, 50ms * 2^attempt))   # jittered backoff
        retry
    return CONFLICT_409                         # surface to client, don't spin

Numbers: monitor occ.conflict / occ.attempts. Below ~5% OCC is nearly free. At 10–20% it's a warning; at >20% the entity is a hot spot and the design should move to a single-writer or split state. DynamoDB also caps a single partition at ~1,000 WCU/sec, so a hot item hits a hard wall regardless of conflict rate.

4. Single-Writer Ownership with Fencing — Kafka Partition + Fenced Store Writes#

The highest-throughput answer to a hot key is to stop sharing it: route every command for a key to exactly one owner, which applies them sequentially from memory.

# Producer: all commands for an event go to the same partition
kafka.produce(topic="inventory-cmds", key=event_id, value=ClaimCmd(user, qty, req_id))

# Owner (one consumer per partition via consumer group assignment)
on_partitions_assigned(parts, generation):
    fence = generation                       # monotonically increasing per assignment
    for p in parts: state[p] = load_snapshot(p) then replay from snapshot offset

on_message(cmd):
    if seen(cmd.req_id): return reply(previous_result)   # idempotent
    inv = state[cmd.event_id]
    result = inv.available >= cmd.qty ? inv.claim(cmd) : SOLD_OUT
    batch.append(result)

every 20ms or 500 results:
    # Fenced write: rejected if a newer owner has taken over
    db.execute("UPDATE owners SET fence = $1 WHERE part = $2 AND fence <= $1", fence, part)
    if rows == 0: abort_and_drop_partition()
    db.bulk_insert(batch); kafka.commit_offsets(); publish_replies(batch)

Why it matters: an in-memory owner can process 100K+ commands/sec per partition because there is no lock and no per-command fsync — the batch is the durability unit. The cost is the failover window (consumer-group rebalance, typically 3–30s) during which that event's commands queue, and the need for the fence so the old owner, if merely paused, cannot write after the new owner starts.

Technique Comparison

Architecture Diagram#

How to narrate it: contention is handled in three rings. The outer ring (bot filter, waiting room) removes contenders who were never going to win. The middle ring (admission tokens) meters the rest to roughly the rate the hot path can absorb. The inner ring (bucketed atomic claims) is where the invariant actually lives — and the durable record behind it is protected by a schema constraint and a reconciler, so a bug in any outer ring degrades fairness, never correctness.

Failure Scenarios#

1. Lock Convoy — Checkout Database Down for 11 Minutes#

A merch store launches a limited drop of 5,000 hoodies. The purchase path uses SELECT ... FOR UPDATE on the inventory row, then calls the payment authorizer, then commits — the lock spans an external call that takes ~300ms.

t=0      Drop opens. 40K requests/sec. Each txn holds the row lock ~320ms.
t=+1s    Throughput ~3 purchases/sec. 39,997 requests waiting on the lock.
t=+5s    Connection pool (400) fully occupied by waiters. Other queries queue.
t=+15s   Order-history, login session writes time out (same cluster).
t=+40s   App retries on timeout; offered load doubles to 80K/s.
t=+3min  Health checks fail; autoscaler adds app hosts -> more connections -> worse.
t=+8min  On-call disables the drop via feature flag; waiters drain.
t=+11min DB recovers. 212 hoodies sold. Social media notices.

Detection: db.lock_wait_seconds p99 > 1s; db.connections.active / max > 90%; pg_stat_activity waiting on one relation; checkout.success_rate collapses while checkout.attempts spikes. Blast radius: the whole database cluster — every service sharing it, not just the drop. Mitigation: kill switch on the sale; cap per-endpoint DB concurrency (bulkhead) so one hot path cannot take the pool. Prevention: never hold a lock across a network call; atomic conditional claim + async payment against a hold; admission layer for any launch with expected demand/supply > 10×; launch review checklist. Owner: commerce team owns the purchase path; DB platform owns per-service connection limits.

🎯 Staff Insight: The bug was not "too much traffic". It was a 300ms external call inside a critical section, which cut the ceiling by ~300×. The first question in any contention review is: what is inside the lock?

2. The Paused Holder — Double Payout from an Unfenced Lock#

A payouts job takes a Redis lock (SET payout:merchant:88 NX PX 30000) before sending a daily payout. One worker hits a 41-second stop-the-world GC pause after acquiring the lock and before calling the bank API.

t=0      Worker A acquires lock (TTL 30s). Begins payout for merchant 88.
t=+2s    Worker A enters full GC (heap misconfigured after a deploy).
t=+30s   Lock expires. Scheduler retries; Worker B acquires lock.
t=+33s   Worker B sends payout $48,200. Marks payout row paid.
t=+43s   Worker A resumes, still believes it holds the lock. Sends payout $48,200.
t=+1d    Reconciliation with the bank flags 37 duplicate payouts, $1.1M.

Detection: payout.duplicate_detected from bank reconciliation (too late); earlier: jvm.gc.pause_seconds > lock TTL; lock.lease_expired_while_held (instrument the client to log when it renews after expiry). Blast radius: every merchant processed by a paused worker during the window — money out the door. Mitigation: freeze payouts; claw back via bank; idempotency key on the bank API call keyed by (merchant, payout_date). Prevention: the side effect itself must be idempotent (bank-side idempotency key) and the payout row update fenced: UPDATE payouts SET state='sent', fence=$token WHERE id=$1 AND state='pending'. The lock becomes an efficiency measure, not the safety mechanism. Owner: payouts team owns idempotency; platform owns the lock library (which should refuse to hand out leases without a token).

3. Retry Storm on a Hot Coupon — OCC Melts the Cart Service#

A "first 10,000 redemptions" coupon is stored as one DynamoDB item with a redeemed count and version. Every cart applies the coupon with an OCC update and retries on conflict up to 10 times with no jitter.

t=0      Coupon goes live in a push notification to 4M users.
t=+20s   2,500 apply attempts/sec on one item. Conflict rate 85%.
t=+30s   Retries push offered writes to ~18K/sec on one partition (cap ~1K WCU).
t=+35s   Throttling (ProvisionedThroughputExceeded) -> more retries.
t=+1min  Cart service threads blocked in retry loops; cart p99 14s. Checkout fails site-wide.
t=+9min  Coupon disabled. 10,000 redemptions took 9 minutes; ~40K users saw errors.

Detection: occ.conflict_rate > 20% on a single entity; ddb.throttled_requests on one partition key; cart.p99 rising with flat traffic. Blast radius: the entire cart service, because retries consumed shared threads. Mitigation: turn off the coupon; cap retries at 3 with jitter. Prevention: limited-quantity coupons use pre-minted redemption tokens (10,000 items, claimed with attribute_not_exists(claimed_by) on a random token) — contention is spread across 10,000 keys; per-feature retry budget in the client library. Owner: promotions team owns the coupon model; platform owns the retry budget defaults.

Operational Reality Matrix#

The Principal Lens#

Why L7 Sees This Problem Differently#

A Staff engineer makes one sale survive. A Principal engineer notices that the company runs drops, coupons, appointment slots, limited inventory and payout jobs — five teams, five hand-built hold/commit/release implementations, five independently discovered leaked-hold bugs — and that every high-severity contention incident in the last two years happened during a planned event that someone knew about a week in advance. At org scale, contention is a scarce-resource primitive plus an event-readiness process. The questions become: should "hold, commit, release, expire" be a platform with one well-tested implementation; which events need a launch review; who has the authority to pause a sale; and how much capacity does the company pre-buy for peaks that last ten minutes.

The Org-Level Fault Line#

A shared inventory/reservation platform vs per-domain contention handling.

The Principal default is the third row: the platform owns atomic claim, hold lifecycle, fencing, reconciliation and the admission service; each domain owns policy — hold duration, per-user limits, fairness rules — and runs in its own cell.

Cost Model#

Assumptions: managed Postgres primary ~$3K/month (large) to ~$15K (largest + replicas), Redis cluster ~$1.5K/month per 4-shard cell, waiting-room/edge compute billed per peak hour, engineer fully loaded ~$25K/month. Peak events last minutes but drive provisioning.

The Principal observation: at growth scale, the expensive line is not infrastructure — it's the incident. A single oversold drop costs more than a year of the admission service. That makes "admission layer + launch review" an insurance purchase that should be justified as such, and it means the right metric for the platform is incident-free events, not cost per request.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Scarce-Resource Contention Standard (v1)

Scope: Any write path that enforces a quantity or exclusivity invariant (inventory, seats, slots, quotas used for billing, payouts, leader election for side-effecting jobs).

MUST:

1. The invariant is enforced by a single atomic operation (conditional write, Lua script, or single-writer owner) and backed by a storage-level constraint where the store supports one.
2. No critical section may contain a network call to another service.
3. Locks that guard side effects MUST issue fencing tokens, and the protected write MUST validate them. Unfenced locks are permitted only for deduplicating effort.
4. Every hold has a TTL, an idempotent release path, and a published holds.expired_unsold metric.
5. Events with forecast demand/supply > 10× or > 5K attempts/sec MUST pass launch review and run behind the admission service with a tested kill switch.

SHOULD: Retry budgets (≤ 3 attempts, jittered) from the platform client; a reconciler comparing fast-path counts to the durable record within 5 minutes of event close.

Exceptions: Approved by the inventory platform owner, time-boxed to one quarter.

Success metrics: zero oversell incidents; stranded inventory < 0.5% of supply at event close; no contention incident affecting a service outside the event's own cell.

What I'd Tell the VP#

Our biggest launches are also our riskiest moments, because thousands of people try to buy the same few items at once. Twice this year that took down checkout for everyone, and once we sold more than we had. I'm proposing a single, shared way to handle limited inventory — a waiting room in front and one tested component that does the claiming — instead of each team building its own. It costs about three engineers for two quarters, which is less than one bad launch costs us in refunds. In exchange, launches become routine: a checklist, a rehearsal and an on-call person with a pause button.

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "I'd use bucketed inventory in Redis with a Lua claim." — Correct for one event; silent on the four other teams with the same problem.
• "We'll add a waiting room for big launches." — Doesn't define who decides what's "big", who owns the pause button, or how the org rehearses.
• "Fencing tokens solve the paused-holder problem." — True, but doesn't make fencing the default in the shared lock library so nobody can forget it.

In the Wild#

Google Chubby: Sequencers for Lock Safety#

Google's Chubby paper (OSDI 2006) describes a coarse-grained lock service built on Paxos and explicitly addresses the paused-holder problem: a lock holder can obtain a sequencer — an opaque byte string describing the lock and its generation — and pass it to the servers it talks to, which check it before acting. The paper also describes a lock-delay fallback for servers that don't check sequencers.

Staff insight: The people who built one of the most widely cited lock services did not trust leases alone. Citing Chubby lets you say "a lock protecting side effects needs a token the resource validates" with a primary source behind it.

Amazon DynamoDB: Conditional Writes as the Contention Primitive#

DynamoDB's documented model for concurrent updates is optimistic: ConditionExpression on PutItem/UpdateItem (e.g., version = :expected or stock > :zero) evaluated atomically at the item, plus TransactWriteItems for multi-item invariants. Its documentation also describes per-partition throughput limits, which is why a single hot item hits a ceiling regardless of table capacity.

Staff insight: The store gives you single-item atomicity cheaply and multi-item atomicity expensively — exactly the gradient this pattern teaches. Fit the invariant into one item when you can; split the hot item when one partition can't take the heat.

The Redlock Debate: Leases Are Not Mutexes#

In 2016 Martin Kleppmann published a public critique of Redis's Redlock algorithm, arguing that any lease-based lock is unsafe for correctness without fencing tokens, because process pauses and clock behavior can let two clients believe they hold the lock. Redis's author published a rebuttal. The exchange is one of the most-cited public discussions of distributed locking.

Staff insight: You don't need to pick a side on Redlock's clock assumptions. The interview-relevant takeaway both sides agree on in practice: if the lock guards effects, make the resource reject stale holders — and make the effect idempotent anyway.

Practice Drill#

Prompt: "We sell limited-edition sneakers. Each drop is 3,000 pairs; last drop drew 400K users in the first minute. Our purchase endpoint locks the SKU row, checks stock, charges the card, and commits. It fell over and we oversold 140 pairs when engineers 'fixed' it by removing the lock. Redesign it."

Staff Interview Application#

How to Introduce This Pattern#

"Before I pick a lock, I want two numbers: what the invariant is, and how hot the hottest key gets. Here the invariant is no oversell, and the hot key sees maybe 50× supply in the first minute. So I'll do two things: keep most contenders out of the database with an admission layer, and make the claim itself one atomic operation that losers fail fast on."

Lead with the invariant, then the heat, then the primitive, then what losers see, then how you know it held.

When NOT to Use This Pattern#

• No shared mutable state: Append-only events (clicks, logs) don't contend — each write is its own row. Don't add locks to make them "safe".
• Approximate is fine: View counts and soft quotas should aggregate locally and flush; coordinating them is cost without value.
• Low heat: Under ~50 writes/sec per key, a short transaction with SELECT FOR UPDATE is the most readable correct answer. Don't bucket a B2B order table.
• The conflict is a product question: Two editors changing the same document is a merge problem — see Collaborative Editing — not a locking problem.

Follow-Up Questions to Anticipate#

Evaluation Rubric#

Strong Hire Signals

Lean No-Hire Signals

Common False Positives: Reciting Paxos or Redlock internals ≠ protecting the invariant. Naming "optimistic locking" ≠ knowing its conflict-rate ceiling. Drawing a queue ≠ explaining who owns the state behind it and what happens on failover.

Capacity Planning Quick Reference#

Sizing the Contended Path#

contention_factor   = peak_writes_per_key × critical_section_seconds   # > 1 means queueing
lock_ceiling        = 1 / critical_section_seconds                      # per key
buckets_needed      = ceil(peak_writes_per_key / (0.5 × single_key_ceiling))
admitted_rate       = remaining_supply × 1.5 / admission_wave_seconds
stranded_at_peak    = concurrent_holds × abandonment_rate
occ_attempts_per_ok ≈ 1 / (1 − conflict_rate)                          # 2.5× at 60%

Key Numbers Worth Memorizing#

Common Pitfalls Checklist#

• The invariant is enforced by one atomic operation, not read-check-write
• No network call to another service inside any critical section
• Storage-level constraint backs the invariant (CHECK, unique key)
• Locks that guard effects issue fencing tokens; effects are idempotent
• Retries are capped (≤ 3) and jittered; losers fail fast
• Hot keys are measured before launch; buckets or single-writer planned for >1K/s
• Holds have TTL, idempotent release, and a stranded-inventory metric
• Admission layer fails closed, and someone on-call owns the pause button