Consistency Models & Partition Behavior

Why This Matters#

Consistency is not a vocabulary quiz. Nobody gets leveled Staff for reciting the difference between sequential and linearizable. Consistency is a question about which anomaly the business can tolerate, what it costs to prevent the others, and who reconciles the damage when the network splits. "Strong consistency" is not a feature you turn on; it's a latency bill paid on every write and an availability bill paid during every partition.

Most candidates reach for one of two defaults. The first: "We'll use a strongly consistent database," with no mention that a cross-region linearizable write costs 60–150ms of consensus per commit and becomes unavailable on the minority side of a partition. The second: "It's eventually consistent," with no bound on how eventual, no statement of what users see in the meantime, and no plan for conflicting writes. Both answers are reasonable. Neither says who pays.

Staff engineers do three things differently. They name the anomaly instead of the model ("a user can post a comment and not see it on refresh" — that's a read-your-writes violation). They scope the guarantee ("linearizable per account, eventual across accounts") instead of choosing one model for the whole system. And they decide partition behavior up front — which operations fail closed, which fail open, and who reconciles afterward — because that's where the CAP theorem stops being a slogan and starts paging people.

The 60-Second Version#

• Name the anomaly, not the model. Stale read, lost update, write skew, read-your-writes violation, causality violation. Business owners understand anomalies; they don't understand "sequential consistency."
• Linearizability costs a round trip to a quorum on every write — ~1–5ms within a region, ~60–150ms across regions. It also means the minority side of a partition stops serving writes (and linearizable reads).
• Most user-facing products need session guarantees, not global linearizability. Read-your-writes + monotonic reads per user covers ~90% of "the user saw stale data" complaints at a fraction of the cost.
• CAP only applies during a partition. PACELC completes it: else (normal operation) you trade latency vs consistency — and that's the tradeoff you pay every second, not once a year.
• Quorums: R + W > N gives overlapping reads and writes (e.g., N=3, W=2, R=2), but it is not linearizability on its own — sloppy quorums, hinted handoff, and last-write-wins clocks all leak anomalies.
• Isolation levels are consistency models for transactions. Postgres defaults to Read Committed; MySQL InnoDB to Repeatable Read. Neither prevents write skew. SERIALIZABLE does, at the cost of aborts you must retry.
• Last-write-wins silently discards data. With clock skew of even 10–100ms, the "last" write is sometimes the earlier one. If losing a write is unacceptable, you need versions, CRDTs, or a single writer.

How Consistency Models Work#

The Spectrum#

From strongest (most expensive) to weakest (cheapest):

Arrows point from stronger to weaker. Everything above causal requires coordination that makes it unavailable on the minority side of a partition; causal and below can stay available. That line — not the CAP acronym — is the practical boundary.

The Four Session Guarantees#

CAP and PACELC#

CAP: during a network Partition, a system must choose between Consistency (linearizability — refuse requests that can't be answered correctly) and Availability (answer every request, possibly with stale or conflicting data).

PACELC: if Partition, choose A or C; Else, choose Latency or Consistency.

Partitions are rare; latency is constant. PACELC's "else" branch is the one you pay for every request, which is why it matters more in design interviews than the CAP branch.

Core Strategies#

Strategy 1: Single Leader, Read from the Leader#

write(key, value):
    leader.write(key, value)                 # leader orders all writes
    replicate_sync_to_quorum(key, value)     # e.g., 1 of 2 followers acks

read_linearizable(key):
    return leader.read(key)                  # plus a lease or read-index check
                                             # so a deposed leader can't answer

When to use: Money, inventory, uniqueness constraints, anything where a stale read causes a wrong decision.

Failure mode: The leader is the throughput ceiling and the failover risk. A leader that doesn't know it's been deposed (GC pause, partition) can serve stale reads unless reads are protected by a lease or a read-index round trip — the classic bug Jepsen has found in more than one database.

Strategy 2: Quorum Reads and Writes#

N = 3 replicas, W = 2, R = 2        # R + W > N → read and write sets overlap

write(key, value, version):
    send to all N; succeed when W acknowledge

read(key):
    ask R replicas; return highest version
    if replicas disagree: read-repair the stale ones

When to use: Leaderless stores where you want tunable consistency per request (Cassandra QUORUM / LOCAL_QUORUM, Riak).

Failure mode: R+W>N is necessary but not sufficient. Sloppy quorums (writes accepted by non-home nodes during failures), concurrent writes resolved by last-write-wins, and a write that reached fewer than W nodes before failing (it may or may not become visible) all produce non-linearizable behavior.

Strategy 3: Session Guarantees via Version Tokens#

write(user, key, value):
    version = primary.write(key, value)      # returns log position / LSN
    return { ok, session_token: version }

read(user, key, session_token):
    replica = pick_replica()
    if replica.applied_version < session_token:
        replica = primary                    # or wait up to 50ms for catch-up
    return replica.read(key)

When to use: Most user-facing products. Users must see their own writes; they rarely notice that other users' writes arrive 500ms late.

Failure mode: Tokens lost across devices or channels. The user writes on the phone and reads on the laptop — a per-device session guarantees nothing across them. Scope the token to the user (stored server-side) if cross-device matters.

Strategy 4: Eventual Consistency with Explicit Conflict Resolution#

When to use: Multi-region active-active, offline-capable clients, high-availability writes.

Failure mode: Choosing LWW by default and discovering months later that concurrent edits were silently discarded. LWW is a data-loss policy — someone in product must sign off on it.

Strategy 5: Pick the Transaction Isolation Level on Purpose#

When to use Serializable: invariants that span multiple rows — "at least one doctor on call," "total allocations ≤ budget," "username unique across two tables."

Failure mode: Turning on Serializable without retry logic. Under contention, serialization failures (40001 in Postgres) can reach several percent of transactions; code that doesn't retry turns them into user-visible errors.

Partition Behavior: The Hard Sub-Problem#

CAP says you must choose during a partition. It doesn't say what to choose, per operation, or who cleans up. That's the design work.

Decide Per Operation, Not Per System#

🎯 Staff Move: "I'm not going to pick CP or AP for the whole system. Payments and seat holds fail closed — if the region can't reach a quorum, those endpoints return 503 and the UI says 'try again.' Cart, comments, and likes fail open and merge on reconnect. Product has to sign off on the list, because they're choosing which failures customers see."

What a Partition Actually Looks Like#

Region A (2 of 3 replicas)            |  Region B (1 of 3 replicas)
--------------------------------------+--------------------------------------
Majority side: elects/keeps leader    |  Minority side: loses quorum
CP ops: continue normally             |  CP ops: time out → 503 after ~1–5s
AP ops: continue                      |  AP ops: accept writes locally
                                      |  Clients: see their own writes; not A's
--- partition heals after 4 minutes ---
Replicas exchange missed writes. CP data: B catches up from A's log.
AP data: concurrent writes to the same key now conflict → merge or LWW.

The partition's real cost is paid after it heals: conflicting writes, duplicated side effects (emails sent twice), and derived data (search indexes, counters, caches) that saw one side's history.

The Anomalies, Concretely#

Invariant: at least 1 doctor on call for shift 42.
T1: SELECT count(*) FROM oncall WHERE shift=42   → 2
T2: SELECT count(*) FROM oncall WHERE shift=42   → 2
T1: DELETE FROM oncall WHERE shift=42 AND doctor='alice'
T2: DELETE FROM oncall WHERE shift=42 AND doctor='bob'
Both commit under Snapshot Isolation → 0 doctors on call.

Failure Scenario: The Async Replica Promotion#

t=0      Primary in us-east-1 serving 8K writes/s; async replica in us-west-2, lag ~200ms.
t=+0s    us-east-1 primary host fails hard (disk controller).
t=+30s   Automated failover promotes the us-west-2 replica.
t=+31s   ~1,600 committed writes (200ms × 8K/s) exist only on the dead primary.
t=+5min  Customers report orders that "disappeared"; some were charged via the payment provider.
t=+2h    Old primary's disk recovered; 1,600 orphaned writes identified.
t=+2d    Manual reconciliation: re-insert non-conflicting rows; refund 37 double-charged orders.

Detection: replication.lag.seconds before failover (it was known); post-failover diff of the old primary's WAL vs the new primary; spike in orders.not_found support tickets. Blast radius: every write committed in the last replication-lag window — here ~1,600 orders. Mitigation: fence the old primary so it can't accept writes if it comes back; reconcile from its WAL. Prevention: semi-synchronous replication to at least one replica before ack (≈ +1–2ms in-region), or make cross-region failover a human decision with an explicit "accept data loss of N seconds" sign-off. Owner: database platform team owns failover policy; the payments team owns the reconciliation runbook, because only they know which lost writes had external side effects.

Operational Reality Matrix#

Visual Guide#

Choosing a Consistency Level per Operation#

Read-Your-Writes with a Version Token#

Behavior During a Partition#

Implementation Patterns#

Fencing Tokens#

Every leader term gets a monotonically increasing epoch. Storage rejects writes carrying an epoch lower than the highest it has seen. A deposed leader waking from a 30-second GC pause can't corrupt data because its token is stale.

Consistency Scoped by Key#

Most systems need strong consistency per entity, not globally. Route all writes for an account to one partition leader; you get linearizability for that account and horizontal scale across accounts. Cross-entity invariants ("transfer between two accounts") then need a transaction protocol or a saga — and that's where the real cost lives.

Commit Wait and Hybrid Clocks#

Spanner waits out TrueTime's uncertainty (ε, typically a few milliseconds) before acknowledging a commit, so timestamp order matches real-time order. CockroachDB uses hybrid logical clocks with a bounded max clock offset (500ms default) and restarts reads that encounter uncertainty. Both convert clock uncertainty into latency, which is the right trade: you pay a few ms instead of getting anomalies.

Outbox for Cross-System Consistency#

A database commit plus a Kafka publish is two systems and no transaction. Write the event into an outbox table in the same transaction; a relay publishes it. Consumers see events at-least-once and in commit order per key. This gives you eventual consistency with no lost events across service boundaries — which is what most "consistency between services" questions are actually asking for.

Quorum Configurations#

When NOT to Pay for Strong Consistency#

• Derived or recomputable data — search indexes, recommendation scores, analytics rollups. Rebuild beats coordinate.
• Data whose staleness no human can perceive — like counts, view counts, "last seen 2 minutes ago."
• Reads that inform, not decide — the product page stock count; the reservation decides.
• Cross-region paths on the user's critical latency budget — 70–150ms per write is usually a product regression the business didn't ask for.

The Numbers in Context#

How This Shows Up in Interviews#

Scenario 1: "Is your system CP or AP?"#

Don't answer with one letter. "Per operation. Balance changes and seat holds are CP — they need a quorum and fail with a retryable error on the minority side. Feed, likes, and comments are AP with session guarantees so users see their own writes. And in normal operation — the PACELC 'else' — I'm choosing latency for reads by serving from local replicas with a 1-second staleness bound, and consistency for writes by going to the leader."

Scenario 2: "Design the inventory service for a flash sale across two regions." (Full Walkthrough)#

Step 1 — Name the anomaly that matters. "The anomaly the business can't tolerate is overselling: two buyers both getting the last unit. Stale reads of the stock count on the product page are fine — nobody is harmed if the page says '12 left' and it's really 9."

Step 2 — Scope the strong guarantee. "So I split reads from reservations. Product-page stock counts are eventually consistent, served from a cache with a 2-second TTL. The reservation — decrementing stock — is linearizable per SKU."

Step 3 — Place the leader. "Cross-region consensus on every reservation costs ~70ms and makes the minority region unable to sell during a partition. Instead each SKU's inventory has a home region leader, and I pre-allocate stock: region A gets 60% of units, region B 40%, based on forecast demand. Each region decrements its own allocation locally — linearizable within the region at ~2ms."

Step 4 — Rebalance allocations. "When one region runs low, it requests more units from the other through a single transfer transaction. That's the only cross-region consistent operation, and it's rare."

Step 5 — Partition behavior. "If the regions partition, each keeps selling from its own allocation — no oversell is possible, because neither region can spend units it doesn't hold. The worst case is that region A sells out while B still has stock. Product signs off that 'sold out a little early in one region' is acceptable during a partition; overselling is not."

Step 6 — Owners. "Inventory team owns allocation policy and the transfer protocol; the flash-sale team owns the demand forecast that sets initial splits."

Why this is a Staff answer: It identifies the one anomaly that matters, applies strong consistency only where that anomaly lives, converts a global consensus problem into local ones through allocation, and gets product to sign off on partition behavior in business terms.

Scenario 3: "Users say their comment disappears after posting, then comes back."#

This is a monotonic-reads plus read-your-writes violation. "The write went to the primary; the next read hit a lagging replica, the one after that hit a caught-up replica. I'd return an LSN session token on write and route reads to replicas that have applied it — or pin the user to the primary for ~5 seconds after a write. I'd also add replication.lag.seconds per replica to the load balancer's health check so a replica lagging past 1s stops receiving reads."

Scenario 4: "Two admins edit the same config simultaneously. One change is lost."#

Lost update. "Every config write carries the version it read — UPDATE config SET … , version = v+1 WHERE id = ? AND version = v. If zero rows update, the client re-reads and shows the conflict. For a config system, I'd rather make the second admin merge explicitly than silently pick a winner."

Advanced Patterns#

The Principal Lens#

Why L7 Sees This Problem Differently#

A Staff engineer picks the right consistency model for each operation in their system. A Principal engineer notices that the anomalies users actually hit rarely live inside one database — they live between services. The order service is linearizable, the payment service is linearizable, and the customer still sees "order confirmed, payment failed" because the two are stitched together by an async event with no contract about ordering, duplication, or staleness. At L7, consistency becomes a cross-team contract: what each service promises about the freshness and ordering of the data it exposes, and which team owns reconciliation when those promises are broken.

The Org-Level Fault Line#

One consistency posture for the whole company vs per-service choice.

The Principal position: a small number of consistency tiers (e.g., Ledger, Transactional, Session, Eventual), each with a paved-road datastore and a published staleness/ordering contract, and a rule that every cross-service data flow declares which tier's contract it offers.

Cost Model#

Assumptions: a regional strongly consistent SQL cluster (3 replicas) ≈ 3× the cost of a single primary with async replicas; a multi-region consensus database ≈ 1.5–3× the regional cluster (more replicas, cross-region transfer, premium pricing); cross-region transfer ~$0.02/GB; engineer ~$25K/month.

The Principal insight: making everything globally consistent roughly doubles the database bill; making nothing consistent staffs a reconciliation team. The tiered model spends consensus dollars only where an anomaly costs more than the consensus.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Data Consistency Tiers (v1)

Scope: All persistent data owned by production services, and every cross-service data flow (APIs, events, CDC).

Tiers: Ledger (strict serializable, multi-region, zero data loss on failover) · Transactional (serializable or linearizable within a region; RPO ≤ 1s cross-region) · Session (read-your-writes + monotonic reads per user; replica staleness ≤ 1s p99) · Eventual (converges within a documented bound; conflicts resolved by a declared policy).

MUST:

1. Every datastore and every event stream declares its tier in the service catalog.
2. Any data involving money movement or legal records MUST be Ledger tier.
3. Last-write-wins MUST NOT be used for user-authored content without a product sign-off recorded in the design doc.
4. Cross-service events MUST be published via transactional outbox or CDC; dual writes are prohibited.
5. Services MUST emit replication.lag.seconds and consistency.violation metrics (e.g., RYW token misses).

SHOULD: prefer scoping strong guarantees per entity over global ones; prefer escrow/allocation over cross-region consensus for numeric budgets.

Exceptions: Architecture review; exceptions carry a named reconciliation owner.

Success metrics: reconciliation incidents per quarter trending to zero; 100% of Ledger data on the Ledger tier within 4 quarters; no dual-write findings in design review.

What I'd Tell the VP#

"Some of our data has to be exactly right — money, inventory, legal records — and some of it can be a second or two behind without anyone noticing. Today every team decides for itself, so we pay for perfect accuracy in places that don't need it and we get mismatches in places that do; finance spent six weeks last year reconciling one of them. We want four clear categories, a standard database for each, and a rule that money always goes in the strictest one. That puts our most expensive database spend on about a tenth of our data instead of all of it. It also shrinks the reconciliation work that currently takes several people."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll use Spanner so we don't have to think about consistency." — Buys correctness inside one database and ignores the boundaries where anomalies actually occur.
• "Payments is CP and feed is AP." — Right classification, no mechanism to make 300 services classify consistently.
• "We'll reconcile with a nightly job." — Names a mechanism with no owner, no SLA, and no plan for side effects that already escaped.

In the Wild#

These are public, documented examples.

Google Spanner: External Consistency via TrueTime#

Spanner (OSDI 2012) provides externally consistent (strictly serializable) transactions across globally distributed data. It uses TrueTime — GPS receivers and atomic clocks in each data center — to bound clock uncertainty, and makes each commit wait out that uncertainty before acknowledging. Replication within each shard is Paxos-based, so writes pay a majority round trip in addition to commit wait.

Staff insight: Spanner doesn't make consistency free; it makes the cost explicit and bounded — a few milliseconds of commit wait plus consensus latency. When you cite it, cite the price: "strict serializability at the cost of a majority round trip on every write."

Amazon Dynamo: Availability First, Merge Later#

The 2007 Dynamo paper described an always-writable store for the shopping cart. It used sloppy quorums and hinted handoff to accept writes during failures, and vector clocks to detect conflicting versions, which were returned to the application to merge. For the cart, the merge was a union of items — which meant deleted items could reappear after a conflict, a known and accepted anomaly.

Staff insight: Amazon chose availability for "add to cart" because a rejected add loses a sale, and accepted a resurrected item as the cost. That's exactly the "name the anomaly, name who pays" reasoning: the customer occasionally removes an item twice; the business never loses an add.

Jepsen: Advertised vs Actual Guarantees#

Kyle Kingsbury's Jepsen project has published analyses of many distributed databases and queues under partitions, clock skew, and process pauses. Repeatedly, it has found systems that violated their documented consistency or isolation guarantees in specific configurations — lost acknowledged writes, stale reads from deposed leaders, and anomalies prohibited by the claimed isolation level. Vendors have fixed many of these issues in response.

Staff insight: A vendor's consistency claim is a hypothesis, not a guarantee. The Staff move is to name the configuration you rely on (write concern, read concern, isolation level) and how you'd test it — not to quote the marketing page.

Staff Calibration#

What Staff Engineers Say (That Seniors Don't)#

Staff Sentence Templates#

"The anomaly the business can't tolerate here is [anomaly], so [operation] is [guarantee]; everything else is [weaker guarantee] with a [bound] staleness bound."

"During a partition, [operation] fails closed and returns [error]; [other operation] accepts writes locally and reconciles by [merge policy]. Product signed off on [user-visible effect]."

"Async failover loses [lag] × [write rate] ≈ [N] writes. [Team] owns reconciliation, because only they know which of those had external side effects."

Key Takeaway: Consistency is priced per operation. Buy linearizability where an anomaly costs more than a quorum round trip; buy session guarantees where users would notice; let everything else be eventual with a stated bound.

Common Interview Traps#

• Choosing one model for the whole system. Scope guarantees per operation or per entity.
• Treating R + W > N as linearizable. Sloppy quorums, LWW, and partial writes leak anomalies.
• Saying "eventually consistent" without a bound. How eventual? 100ms? An hour? What do users see meanwhile?
• Assuming the default isolation level prevents write skew. Read Committed and Snapshot Isolation don't.
• Using wall-clock LWW across regions. Clock skew reorders writes; you lose data silently.
• Dual writes to DB and queue. One succeeds, one fails, and they diverge. Use an outbox.
• Ignoring the healed partition. The hard part is reconciliation and side effects, not the partition itself.
• Trusting vendor claims without naming the configuration. Write concern and read concern matter.

Practice Drill#

Prompt: "We're expanding from one region to two, active-active. Product wants users to be able to edit their shared documents from either region with low latency. Walk me through consistency."

Where This Appears#

• Replicated Data Store — Quorums, leader election, replica lag, and failover data loss in depth
• Payment Processing — Ledger-grade consistency, idempotency, and reconciliation across providers
• Reservation Systems — Write skew, holds, and fail-closed booking
• Flash Sales & Ticketing — Escrow/allocation of inventory across regions
• Collaborative Editing — CRDTs vs OT and multi-region active-active editing
• Distributed Consensus — Raft/Paxos, fencing tokens, and linearizable reads
• Database Selection — Matching consistency guarantees to datastore choices

Related Foundations & Patterns: Consistent Hashing · Sharding & Partitioning · Dealing with Contention · Distributed Coordination

Related Technologies: PostgreSQL · Cassandra · DynamoDB · ZooKeeper & etcd