Technologies referenced in this case study: Redis · PostgreSQL · Apache Kafka · DynamoDB

Related: Caching Fundamentals · Consistent Hashing · Scaling Reads · CDN & Edge Caching · Replicated Data Store · Multi-Region Active-Active · Circuit Breakers · Degraded Mode Framework · Consistency Models

How to Use This Case Study#

This case study assumes you already know what cache-aside and LRU are — Caching Fundamentals covers those. Here the subject is the cache as a distributed system you operate: its correctness contract, its failure modes, and the database it is quietly holding up.

Without load-bearing awareness (cache treated as optional):
t=0:       Ops starts a rolling upgrade of the 12-node cache cluster, 1 node per 2 minutes.
           Each restarted node comes back empty. Hit rate 97% → 89% after 3 nodes.
t=+6min:   DB read QPS: 9K → 36K. DB was provisioned for 15K. p99: 4ms → 900ms.
t=+8min:   App threads block on DB. API p99 → 6s. Health checks fail; app pods recycle,
           losing their in-process caches too. Hit rate 81%.
t=+10min:  DB connection pool exhausted. Full outage of product pages.
t=+45min:  Upgrade paused, traffic shed at the edge, DB recovers, cache refills over 30 min.

With the cache treated as a capacity tier:
t=0:       Upgrade tooling checks DB headroom: at 97% hit rate the DB can absorb loss of
           at most 1 node (≈ +8% of reads) safely. Upgrade proceeds 1 node at a time.
t=+0:      Before restart, node N's hot keys are pre-copied to its replacement (warm handoff);
           reads for N's keyspace go to the replica pool during the swap.
t=+3min:   Hit rate dips 97% → 95.8%, recovers within 90s. DB peak 13K QPS, inside budget.
t=+40min:  12 nodes upgraded. No customer-visible impact. Tooling records the minimum hit
           rate and peak DB QPS for the change log.

Executive Summary

If you only read one section, read this. The rest of the case study expands these pages.

What This Interview Actually Tests#

A distributed cache is not a performance question. Everyone knows RAM is faster than disk.

It is a dependency-and-correctness question: you are creating a second copy of data with weaker guarantees, and the system will start depending on that copy in ways that are invisible until it disappears. It tests:

• Whether you know which job the cache is doing — trimming latency, carrying load the database cannot, or avoiding expensive recomputation — because each one fails differently
• Whether you can state a staleness contract (how wrong, for how long, who agreed) instead of "eventually consistent"
• Whether you know the database is now sized for the miss rate, and design for the day the hit rate drops
• Whether you can keep the cache correct when there are many writers, many readers and an unreliable network in between

The key insight: The moment a cache absorbs enough reads that the database could not serve them alone, it stops being an optimization and becomes a capacity tier with no durability. From then on, cold starts, node loss and hot keys are availability events for the database. Staff candidates notice the crossover point and design warmth, headroom and miss-path protection as first-class features.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll assume this is a load shield: the read path is around 500K reads per second and the database is provisioned for the misses, not the total. That's the version where the cache's failure modes become the database's outages, so it's where the interesting tradeoffs are. If it's only trimming latency, the design gets simpler and I'll say which parts I'd drop."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: These designs are publicly documented, and each one solves a problem interviewers will steer you toward.

Facebook — Memcache at Global Scale#

Facebook's NSDI 2013 paper, Scaling Memcache at Facebook, describes a look-aside cache serving billions of requests per second. Several mechanisms from it are now standard vocabulary: leases — a token handed out on a miss that both throttles concurrent fills (thundering herd) and rejects a set that arrives after the key was invalidated (stale sets); mcrouter, a proxy that routes, batches and replicates requests; a gutter pool of spare servers that absorbs traffic for failed nodes so their load doesn't hit the database; and invalidations driven by tailing the database's commit log and broadcasting deletes, rather than trusting every application write path to delete correctly.

Staff insight: Leases solve two of the hardest interview follow-ups — miss storms and the delete/fill race — with one mechanism. Naming them and explaining both effects is a strong signal.

Netflix — EVCache, Replicated Across Zones#

Netflix built EVCache on Memcached, with a client that writes each item to a copy of the cache in every availability zone and reads from the local zone, falling back to another zone on a miss or failure. Netflix has described it as handling very large request volumes across regions, with cross-region replication for some use cases and warming tools to fill new clusters from existing ones before they take traffic.

Staff insight: Zone-replicated writes trade memory (one copy per zone) for the ability to lose a whole zone's cache without a miss storm against the database. That's the load-shield posture made explicit — pay for redundancy so the database never sees a cold cache.

Uber — CacheFront, Integrated Cache With CDC Invalidation#

Uber has written about CacheFront, a caching layer integrated into its Docstore database's query engine, serving tens of millions of reads per second from Redis. Rather than relying on each service to invalidate correctly, it uses Docstore's change-data-capture stream to invalidate or refresh cached rows after writes commit, and it measures cache staleness directly by comparing samples of cached values against the database.

Staff insight: Two ideas worth stealing: put invalidation where every write passes (the change log), and measure staleness instead of assuming it. "I'd sample 0.1% of cache hits, compare against the DB asynchronously, and alert on the mismatch rate."

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: Reads go near cache → remote cache → read replicas, with at most one in-flight fill per key per pod. Writes never touch the cache directly; the commit log drives invalidation, so every write path — services, migrations, admin scripts — is covered. The spare pool keeps a failed shard's traffic off the database. The two numbers on the dashboard that matter most: db.read_qps against its budget (the cache's real SLO) and cache.stale_sample_rate (whether the staleness contract holds).

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

Forty-five minutes, one cache, and a database you must not hurt. The timing below puts most of the depth on staleness and on the miss path, where interviewers look for judgment.

Phase 1: Requirements & Framing (2–3 minutes)#

Say this, close to verbatim:

"A cache can be doing one of three jobs: trimming latency on reads the database could serve anyway, carrying read load the database can't, or saving expensive recomputation. They fail differently — in the first, losing the cache makes us slower; in the second, it takes the database down. So: how much read traffic, how much of it can the database serve on its own, and how stale can each kind of data be?"

Then commit to numbers:

🎯 Staff Move: "Here's the number that matters: the DB can serve about 60K reads a second, and we have 500K. So the cache is load-bearing — below an 88% hit rate we're down. I'll treat hit rate as a capacity SLO with a floor of 94%, and design cold start, node loss and hot keys around protecting that floor."

Phase 2: Core Entities & API (1–2 minutes)#

Namespace      { name: "product", owner_team, staleness_budget_s, ttl_s, max_memory_gb,
                 invalidation: cdc|app|ttl_only, near_cache: auto|off, value_schema_version }
CacheKey       = "{namespace}:v{schema_version}:{entity_id}[:{variant}]"
CacheEntry     { value, version (source row version or commit LSN), written_at, ttl }
FillLease      { key, token, issued_at }      // issued on miss; invalidated by delete
Invalidation   { namespace, entity_id, source_lsn, emitted_at }

Client API the services see — note what's missing: there is no set exposed to application code for load-shield namespaces.

get(ns, id, loader, opts) -> value
    // near cache → remote → (lease) → loader() → versioned set
    // opts: max_staleness_s, bypass_cache (read-your-writes), timeout_ms
get_many(ns, ids, loader_batch) -> map
invalidate(ns, id)          // app-driven; CDC is preferred

"Applications pass a loader instead of calling set themselves. That's how the client can coalesce misses, enforce leases and attach versions — the safe protocol is the only protocol."

Phase 3: High-Level Architecture (≤5 minutes)#

"Reads go near cache, remote cache, replicas. Writes go only to the database, and the commit log drives invalidation. Every arrow on the miss path has a concurrency limit, because that's the arrow that kills the database."

Phase 4: Transition to Depth (1 minute)#

"The boxes are standard. Three things decide whether this works in production: how we keep entries from staying wrong — the invalidation path and the delete/fill race; what happens to the database when the cache loses data — cold starts, node loss, hot keys; and how many teams share this cluster. I'd like to start with correctness, then the miss path. Sound right?"

Phase 5: Deep Dives (25–30 minutes)#

Deep dive A — correctness (10 min). Staleness budgets per class; the delete/fill race on a sequence diagram; leases or versioned sets; why CDC beats app deletes when there are many writers; read-your-writes for the author via a short-lived per-user bypass marker.

"The race is: reader misses and reads v1, writer commits v2 and deletes, reader sets v1. With leases, the delete invalidates the reader's token, so its set is rejected. Without them, v1 lives for the full TTL."

Deep dive B — the miss path (10 min). Arithmetic of hit rate vs DB load; miss coalescing per key; cache-down posture (50ms timeouts, breaker, shed non-critical reads, serve stale where allowed); warm handoff for planned node replacement; spare pool for unplanned loss; hot-key detection and near-caching.

Deep dive C — tenancy and cost (5 min). Per-namespace memory quotas or pool separation; eviction interference; cost per GB vs DB cost saved.

Phase 6: Wrap-Up (2–3 minutes)#

"To recap: the cache is load-bearing, so hit rate is a capacity SLO with a floor of 94%, and the miss path is protected by coalescing, breakers and a spare pool. Correctness comes from CDC-driven invalidation, lease tokens against the delete/fill race, and per-class staleness budgets product signed. Hot keys are absorbed by a near cache. Next I'd build sampled staleness measurement, per-namespace quotas for the shared cluster, and the cross-region invalidation path for the second region."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Caching is where small, reasonable local decisions accumulate into an unowned global risk. One team adds a cache to fix a slow page; a year later the database has been right-sized to the miss rate; two years later nobody can restart the cluster during business hours. The question tests whether you see the cache as a dependency that changes the system around it: it changes how big the database is, what consistency users see, what an upgrade costs, and who must be in the room when it fails.

It also tests correctness under concurrency with nothing exotic — no consensus, no transactions — just two copies of data and an unreliable ordering between writes, deletes and fills. Candidates who can reason precisely about that race usually reason well about every other distributed race.

1.2 The L5 vs L6 Contrast — Visual#

The Senior path is correct and reactive. The Staff path computes the dependency first — the hit-rate floor — and every later decision protects it.

1.3 The Staff Question That Cuts Through Everything#

"If this cache were empty right now, what would happen — and how wrong is it allowed to be when it's full?"

The first half finds the capacity dependency: if the answer is "the database falls over", every operational procedure — upgrades, failovers, deploys, flushes — must be designed around warmth. The second half forces the staleness contract: if nobody can say "how wrong", nobody can say whether the invalidation design is good enough, and the first stale-price incident will be argued rather than diagnosed.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Intent 1: Latency shield. The database can serve every read, at a cost in latency: say 8ms p50 and 60ms p99 against a 1ms cache hit. The cache improves user experience but nothing breaks without it. These caches are usually small, have moderate TTLs, and can be flushed whenever convenient. The main risk is correctness drift — stale data nobody notices because the cache is "just an optimization". The design can stay simple: cache-aside, TTLs, app-driven deletes.

Intent 2: Load shield. The database was sized — or has been allowed to shrink — to serve the misses. At 500K reads/s and a 60K reads/s database budget, the cache is required infrastructure. Its risks are all availability risks: cold start after a deploy or flush, a shard failure, a hot key exceeding a node's network, a mass expiry of keys that were all written at the same time. The design must include replicas across AZs, miss coalescing, spare capacity for failed nodes, warm handoffs, and a shed path for when all else fails.

Intent 3: Compute shield. The cached thing is expensive to produce: a rendered page (50ms of CPU), a personalized feed (300ms fan-in across 40 services), an ML feature vector, a model response that costs real money per call. Entries are larger (10 KB–1 MB), staleness tolerances are generous (minutes to hours), and the risk is cost — an eviction storm or a version change that forces recomputing millions of entries. Design leans on long TTLs, refresh-ahead for popular entries, size-aware eviction, version-tagged entries so a model change can roll out gradually, and sometimes SSD-backed storage because RAM is too expensive for the working set.

2.2 When NOT to Add a Distributed Cache#

• When the database can serve it with an index or a replica. A missing index or an N+1 query is a cheaper fix than a cache that must be kept correct forever. Profile first.
• When reads are not repeated. Long-tail access (each key read once a day) yields hit rates under 30%; you pay for RAM and invalidation and get little. Check reuse distance before building.
• When correctness can't tolerate any staleness and invalidation can't be made synchronous. Account balances, inventory at checkout, permission checks after revocation — read from the source, or cache with version checks that fall through on doubt.
• When an in-process cache is enough. If the working set fits in ~100 MB and staleness of a few seconds is fine, a per-pod cache has no network hop and no cluster to operate.
• When a CDN can do it. Public, cacheable-by-URL content belongs at the edge — see CDN & Edge Caching.

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: Who Keeps the Cache Honest#

Every cached entry is a claim that "this is still true". Something must retract the claim when the source changes.

Staff default: cache-aside with lease tokens on fill, TTL as a backstop (sized to the staleness budget × ~10, not as the primary mechanism), and CDC-driven deletes once there is more than one writer or any out-of-band write path. App-driven deletes remain as a fast path for the author's own writes.

When to deviate: for data written by exactly one service with no batch paths, app-driven deletes plus leases are enough — CDC is a pipeline you'll have to run. For compute-shield caches, version-tag entries (model_v17) instead of invalidating: a new version is a new key, and old entries age out.

🎯 Staff Move: "Invalidation should follow the writer, not the reader. If I only delete from the service's write path, the first batch job that updates the table directly will create stale entries nobody can explain. So I'd invalidate from the commit log, and the team that owns the table owns that pipeline's SLO."

3.2 Fault Line 2: Staleness Budget vs Hit Rate#

TTL is the knob that trades wrongness for database protection. Turning it is a product decision disguised as a config value.

The arithmetic product needs to see. If a key is written once a day and read 10,000 times a day, a 60-second TTL produces ~1,440 misses/day for that key; a 1-hour TTL produces 24. With reliable invalidation, the TTL can be long and the real staleness is the invalidation lag (< 1s). Without it, staleness is the TTL. So the investment in invalidation buys hit rate — that's the argument that funds the CDC pipeline.

Staff default: per-namespace staleness budgets in a registry; TTL derived from the budget when invalidation is TTL-only, or set long (hours) with CDC invalidation and a sampled staleness monitor proving the budget holds.

When to deviate: data that must never be stale in a specific flow — the price at checkout, a permission after revocation — bypasses the cache in that flow, or reads the cached value with its version and confirms the version against the source. Cache it for browsing; verify it at the moment of commitment.

🎯 Staff Move: "I'd ask product to sign three numbers: price can be 5 seconds stale on browse pages and 0 at checkout; display names 60 seconds; recommendations an hour. Those numbers decide the TTLs and where we bypass. Without them we're guessing, and we'll argue about every stale-data bug."

3.3 Fault Line 3: Optional vs Load-Bearing#

The most important property of a cache is whether the system survives its absence. Most teams find out during an incident.

The math to say out loud. DB load = read QPS × miss rate. At 500K reads/s: 94% hit → 30K/s to the DB; 88% → 60K/s, the limit; 80% → 100K/s, overload. Losing one of 24 shards with no replica and no spare pool drops hit rate by ~4 points → +20K/s. Losing an AZ's third of the cluster with no cross-AZ replicas drops it ~31 points → outage.

Staff default: declare the posture. For load shields: replicas across AZs, a spare pool sized for one failed shard, fill budgets per pod (a token bucket on DB fills, e.g., 200/s per pod × 300 pods = 60K/s ceiling — the DB budget), stale-on-error for classes that allow it, and a shed path for the rest.

When to deviate: if the DB is cheap to over-provision (a managed key-value store such as DynamoDB with on-demand capacity), sizing the DB for full load and keeping the cache optional can be cheaper than engineering a load-bearing cache. Do the price comparison.

🎯 Staff Move: "The fill path gets its own rate limit, sized to the DB's budget. If the cache fails, the worst the application can do to the database is the budget — beyond that, we serve stale or degraded responses. The cache can fail; the database must not fail because of it."

3.4 Fault Line 4: Where the Bytes Live#

Near-cache arithmetic. A key read 600K/s across 300 pods is 2,000 reads/s per pod. With a 2-second near-cache TTL, each pod fetches it once per 2s: 300 pods × 0.5/s = 150 remote reads/s instead of 600K. The cost: up to 2 seconds of extra staleness for that key, and a few KB of memory per pod.

Staff default: remote cluster for everything; near cache only for keys the client detects as hot (above ~1% of a pod's reads or a fixed rate threshold), with a 1–5s TTL that counts against the namespace's staleness budget. Namespaces with zero staleness tolerance opt out.

When to deviate: small, read-mostly reference data (feature flags, currency tables, country lists, under ~100 MB) belongs fully in-process with push-based refresh — no remote tier at all.

🎯 Staff Move: "Consistent hashing gives every key a home; it doesn't stop one key from being popular. For hot keys I'd rather spend 2 seconds of staleness in a near cache than rebuild the cluster topology."

3.5 Fault Line 5: Shared Cluster vs Dedicated#

Why quotas are hard in a shared cluster. Redis and Memcached evict across the whole instance; there is no per-prefix memory limit. Real isolation means separate instances (or separate Redis databases on separate processes), not key prefixes. A "quota" enforced by convention is a request, not a limit.

Staff default: load-bearing namespaces get their own pool; latency-shield namespaces share a pool with per-tenant memory budgets enforced by placing tenants on separate instances within it; the platform owns placement, the client, upgrades and dashboards.

When to deviate: at small scale (< 100 GB total, a handful of services), one shared cluster with good dashboards is fine — the interference risk is lower than the cost of operating many clusters.

🎯 Staff Move: "Prefixes aren't isolation. If another team's batch job can evict our hot set, our database's capacity depends on their release schedule. Load-bearing namespaces get their own pool."

4. Failure Modes & Operational Reality#

4.1 The Cold Restart — A Deploy That Emptied the Cache#

t=0:       A cache client library upgrade changes the key serializer (hash of a struct
           now includes a new field). Every key the new pods compute is different.
           Effective hit rate for upgraded pods: 0%.
t=+2min:   Canary at 5% of pods. DB read QPS 30K → 38K. Within budget; canary "healthy".
t=+10min:  Rollout to 50%. DB QPS 30K → 165K. Replica CPU 100%, p99 3ms → 2.4s.
t=+11min:  App pods time out on fills and retry — fills double. Old pods' hits keep half
           the traffic alive. Error rate 22%.
t=+14min:  Page: db.read_qps > budget, api.error_rate > 5%. On-call rolls back.
t=+16min:  Rolled-back pods hit the old keys — still warm. Recovery in ~2 minutes.
Total:     6 minutes of severe degradation. Root cause: a key change is a cache flush.

Detection: cache.hit_rate per deploy version (not just fleet-wide) — the canary's hit rate dropped to 0% at t=+2min, which a per-version panel would have shown; db.read_qps vs budget.

Mitigation: fill-rate budget per pod caps DB load at the budget no matter how cold the cache is; stale-on-error and shed paths take the excess.

Prevention: key schema version is explicit (ns:v3:id) and changing it requires a migration plan: dual-read (new key, fall back to old key, write new), warm the new keyspace from the old, then cut over. Canary analysis includes per-version hit rate with a hard gate (e.g., canary hit rate within 2 points of baseline).

Owner: caching platform (client library and canary gates); service team for its key schema.

4.2 The Celebrity Key — One Key, One NIC#

t=0:       A live event starts. Key event:9921:summary (3 KB) goes from 2K to 450K reads/s.
           It lives on shard 14.
t=+20s:    Shard 14 network egress: 450K × 3 KB ≈ 1.35 GB/s ≈ 10.8 Gbps on a 10 Gbps NIC.
           Packet drops; every key on shard 14 (~4% of keyspace) times out.
t=+40s:    Clients treat timeouts as misses → fill from DB. Coalescing is per pod, so 300
           pods each fill event:9921 and every other shard-14 key. DB QPS +35K.
t=+60s:    Breaker opens for shard 14 in clients. Shard 14's keyspace now all misses.
t=+3min:   On-call identifies the key from the server's hot-key sampling and hand-copies
           it to a near-cache allowlist via config push. Load drops in 30 seconds.

Detection: client-side hot-key sampler (count top keys per pod over 10s windows, report keys above 0.5% of reads); cache.node_network_bytes vs NIC capacity; cache.shard_timeout_rate by shard.

Mitigation: automatic near-caching for keys above a threshold (no human in the loop); replicate the hottest keys under N suffixes across shards; a shard that times out routes to the spare pool rather than straight to the DB.

Prevention: for scheduled events, pre-mark event keys as hot (the product knows when the final starts); size NICs for the hottest-key scenario, not the average; keep large values out of hot keys (split summary from detail).

Owner: caching platform (client auto-near-cache, hot-key detection); event product team for pre-marking.

4.3 The Stale Fill — A Price That Wouldn't Update#

t=0:       Merchant updates price 49.99 → 39.99. Writer commits to primary, deletes key.
t=-40ms:   (Concurrently) A reader missed, read from a replica lagging 300ms → 49.99.
t=+260ms:  Reader sets price=49.99 with TTL 24h, after the delete.
t=+1h:     Merchant reports the sale isn't showing. Support says "cache, try again later".
t=+6h:     340 orders at 49.99 shown, charged at 39.99 (checkout reads the source). Shoppers
           who saw 49.99 didn't buy; the merchant's promotion underperformed.
t=+24h:    TTL expires; price correct. Ticket closed "resolved itself".

Detection: sampled staleness — for 0.1% of cache hits, asynchronously read the source row and compare versions; cache.stale_sample_rate per namespace with alert above the budget (e.g., > 0.01% of samples older than 5s for prices).

Mitigation: delete the key (or the namespace for that merchant) via an admin tool; never "wait for TTL" on a reported stale value.

Prevention: lease tokens so a fill that started before the delete can't land after it; versioned sets (store the source row version; reject sets with older versions); CDC invalidation applied after replica lag — or a second, delayed delete ~1–2s after the first, longer than p99 replica lag. TTL sized to the staleness budget × ~10 as a backstop, not 24h.

Owner: the service team owning price data; caching platform for the lease protocol in the client.

4.4 The Neighbor's Batch Job — Eviction Interference#

Mon 02:00: Analytics team's new backfill writes 280 GB of per-user aggregates into the
           shared cluster (TTL 7 days, 40 KB values). Cluster memory: 85% → 100%.
02:10:     LRU evicts older keys — including the session and catalog namespaces' hot set,
           which were "older" only because they're long-lived.
02:15:     Catalog hit rate 97% → 79%. Catalog DB QPS 6× normal. It's 2 a.m.; DB copes.
08:30:     Morning traffic ramps. Catalog DB saturates. p99 product page 4s.
09:05:     Page. On-call sees hit rate drop; doesn't see why — no per-namespace memory view.
10:20:     Root cause found by scanning key prefixes. Backfill keys deleted. Recovery 11:00.

Detection: per-namespace memory and eviction counts (cache.evictions{namespace}); cache.hit_rate{namespace} with alerts per load-bearing namespace; cache.bytes_written{namespace} anomaly detection.

Mitigation: delete or TTL-shorten the offending namespace; temporarily raise fill budgets for affected services if the DB has headroom.

Prevention: load-bearing namespaces on dedicated pools; shared pools enforce per-tenant memory by instance placement; new tenants above 10 GB require a capacity review; large-value bulk loads go to a compute-shield pool (or an SSD-backed store), not the hot-path cluster.

Owner: caching platform (placement and quotas); analytics team for their backfill; the incident review includes both.

4.5 Invalidation Lag — The Pipeline Nobody Watched#

t=0:       A schema change adds a column to the orders table. The CDC consumer's
           deserializer fails on the new column type for 0.3% of rows and retries forever
           on the first bad record — partition 7 stalls.
t=+5min:   Invalidations for 1/32 of entities stop. No errors user-side.
t=+2h:     Support tickets: "I changed my address and it shows the old one" — read-your-
           writes bypass covers the author for 10s, then the stale entry returns.
t=+3h:     Engineer notices consumer lag on partition 7 = 3h. Skips the record, lag drains.
           Entries written in the gap remain stale until TTL (6h).

Detection: inval.lag_ms per partition with alert at p99 > 5s; inval.consumer_errors; sampled staleness (4.3) catches it independently of the pipeline's own metrics.

Mitigation: dead-letter poison records after N retries instead of blocking the partition; on lag recovery, issue a namespace-wide (or time-window) bulk invalidation for entities written during the gap.

Prevention: schema changes run through the CDC consumer's contract tests; the pipeline has an owner and an SLO (p99 < 1s); DLQ has an owner who reviews within a business day.

Owner: the data's writer team owns the invalidation pipeline's correctness; the platform owns the CDC infrastructure.

4.6 Operational Reality Matrix#

🎯 Staff Insight: The cache's loud failures — a node dies, the hit rate drops — are the easy ones. The expensive ones are quiet: a stale price that "resolves itself" after 24 hours, an invalidation partition stuck for three hours. "I don't trust a cache whose staleness I'm not measuring. Sampled comparison against the source is the only metric that tells me the contract holds."

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Redis internals fluency ≠ cache design. Knowing about RDB vs AOF, skiplists and cluster slots is useful; if it doesn't lead to a staleness contract and a miss-path plan, it's trivia.
• "Cache invalidation is one of the two hard problems" ≠ insight. Quoting the joke is not the same as naming the race and closing it.
• High hit-rate targets ≠ good design. "We'll get 99.9%" without asking about reuse distance and working-set size is optimism, not engineering.
• Many cache tiers ≠ sophistication. Browser + CDN + near + remote + DB buffer pool, each with its own TTL, often means total staleness nobody can compute.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Eviction algorithm details — "LRU with scan resistance" and move on.
• Redis vs Memcached feature comparison — one sentence: data structures and replication vs multithreaded simplicity.
• Persistence (RDB/AOF) tuning — for a cache of a DB, persistence is usually off or replica-only; say why.
• Building your own cache server — use Redis or Memcached; the design is in the client and the operating model.
• Distributed transactions between cache and DB — they don't exist in practice; say so and move to leases and CDC.

6.4 Follow-Up Questions to Expect#

1. "A user updates their profile and sees the old one. Walk me through every way that can happen."
2. "The cache cluster restarts. What does the database see in the first 60 seconds?"
3. "How do you size the cluster? What determines the number of nodes?"
4. "One key gets 30% of all reads. What happens and what do you do?"
5. "Three services and a nightly batch job write to the same table. How is the cache invalidated?"
6. "How do you change the cached value's format without an outage?"
7. "How do you know your cache is serving correct data right now?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Our product catalog service is slow and the database is struggling. Design a distributed cache for it."

Drill 2: The Hit-Rate Arithmetic#

Prompt: "Your hit rate dropped from 96% to 92%. Is that a big deal?"

Drill 3: Make It Concrete — Size the Cluster#

Prompt: "How many cache nodes do you need?"

Drill 4: The Cache Is Down#

Prompt: "The entire cache cluster is unreachable. Walk me through the next five minutes."

Drill 5: The Hot Key#

Prompt: "During a product launch, one item's key receives 40% of all reads."

Drill 6: Many Writers#

Prompt: "Three services and a nightly batch job update the products table. How is the cache invalidated?"

Drill 7: The Shared Cluster#

Prompt: "Twelve teams share one large cache cluster. Two teams' hit rates dropped overnight. What's going on and what do you change?"

Drill 8: Build vs Buy#

Prompt: "Should we run Redis ourselves on VMs or use a managed cache?"

Drill 9: Changing the Value Format#

Prompt: "We need to change the cached product object's serialization format. How do you roll it out?"

Drill 10: Multi-Region#

Prompt: "We're adding a second region with its own replicas. How does caching work?"

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Incident — The Midnight Drop#

Context: A limited sneaker release goes live at midnight. Traffic to the product page jumps from 20K to 900K reads/s in 90 seconds. Cache hit rate stays at 97%, yet product pages time out for 40% of users and the catalog database is at 100% CPU. The on-call has added cache nodes, which didn't help. You're pulled in.

Questions to Surface First:

• If the hit rate is 97%, where is the DB load coming from — which queries, from which callers?
• Is the release product's key on one node? What is that node's network utilization?
• Are timeouts being treated as misses — so cache slowness converts into DB load?
• What changed in the release flow — any keys with short TTLs, like stock counts?

Typical L5 Approach: Adds more cache nodes and more DB replicas. New cache nodes reshuffle part of the keyspace (consistent hashing moves ~1/N of keys), causing a fresh wave of misses; replicas take 20 minutes to provision. The drop is over before they help.

Staff Approach: Finds two interacting problems. The product key (8 KB with all size variants) lives on one node pushing ~9 Gbps; that node's timeouts are counted as misses by clients, so 300 pods fill the same key from the DB repeatedly — per-pod coalescing doesn't help across 300 pods. Separately, the stock count has a 1-second TTL and 900K reads/s, so it misses ~300 times a second per pod. Fix: push the product key to the near-cache allowlist (config, no deploy); serve stock from a dedicated counter key with a near-cache TTL of 1s and an explicit "stock may lag 1s" label on browse pages; checkout continues to read the source.

Principal Approach: Treats it as a missing process: launches with known hot items must be pre-registered so the platform pre-marks keys hot, pre-sizes, and runs a load test with the actual value sizes. Also changes the client default: a timeout from a single overloaded node routes to the spare pool or serves stale, never directly to the DB.

Key Takeaway: "A high hit rate can coexist with a database meltdown when one key's misses — or timeouts disguised as misses — are concentrated. Look at where DB load comes from, not just at the hit rate."

What clears the Staff bar:

• Looks past the average hit rate to per-key load and node network
• Spots timeouts-as-misses as the amplifier
• Fixes with config (near-cache allowlist), not more nodes

Deep Dive 2: Silent Failure — Three Days of Wrong Permissions#

Context: A customer's security team reports that an employee who was removed from a workspace three days ago could still read its documents for about 70 minutes after removal — the audit log shows reads. Nobody noticed internally. Permission checks are cached with a 1-hour TTL and app-driven invalidation.

Questions to Surface First:

• What wrote the membership change — the main app, an admin API, an SCIM sync job?
• Does that write path delete the permission cache key? Which key — user-scoped, workspace-scoped, or both?
• Is there a delete/fill race window with replica lag?
• How many other removals took the same path, and over what period?

Typical L5 Approach: Shortens the permission TTL to 5 minutes and adds a delete call to the SCIM job. Fixes this path; the next new write path repeats the bug, and DB load for permission checks rises 12×.

Staff Approach: Finds the SCIM sync job writes membership directly via a bulk SQL update, bypassing the service that invalidates. Every SCIM-driven removal for 8 months stayed authorized for up to the TTL plus one race window. Fix: CDC-driven invalidation from the membership table, covering all write paths; permission entries carry the membership row version; revocation-sensitive checks (document reads after a membership change) verify the version with a cheap lookup. Notify affected customers per the security incident process.

Principal Approach: Reclassifies permission data: authorization decisions are a correctness-critical cache with a revocation SLO (e.g., ≤ 5 seconds), not a performance cache. Establishes a rule that any cache of authorization state must invalidate from the source's change stream and report revocation latency as a security metric reviewed by the security team.

Key Takeaway: "Caches of authorization state are security controls. Their staleness budget is a revocation SLO, and their invalidation must cover every write path."

What clears the Staff bar:

• Finds the bypassing write path rather than just shortening TTL
• Moves invalidation to the change stream and adds version checks for sensitive reads
• Creates a synthetic revocation probe — measured correctness, not assumed

Deep Dive 3: Large-Customer Onboarding — The ML Team Wants 4 TB#

Context: The ML platform team wants to serve real-time features (user embeddings and aggregates, ~2 KB each, 2B users) from the shared cache cluster that currently holds 600 GB for twelve services. They need p99 under 5ms at 300K reads/s and plan to refresh all entries every 6 hours via a batch job.

Questions to Surface First:

• What's the access distribution — do all 2B users' features get read, or a recent-active 200M?
• What happens on a miss — is there a source to fill from, or is the batch job the only writer?
• What's the cost of serving a stale or default feature vector — degraded ranking, or broken product?
• How does the 6-hourly refresh write — 4 TB in a burst?

Typical L5 Approach: Adds 70 nodes to the shared cluster to fit 4 TB. The 6-hourly bulk write churns memory and evicts other tenants; the cluster's upgrade and failover procedures now involve a tenant whose data can't be refilled on miss.

Staff Approach: Recognizes this isn't a cache — there's no source to fill from on a miss; it's a feature store serving tier. Different intent, different system: a dedicated pool (or an SSD-backed key-value store, since 4 TB of RAM is ~$20–60K/month while NVMe-backed storage meets 5ms p99 at a fraction of the cost), loaded by bulk swap — write a new version of the dataset, flip a pointer — not by overwriting live keys. Recently active users (~200M, ~400 GB) can sit in RAM; the long tail on SSD.

Principal Approach: Uses the request to define the platform's tenancy rules: the shared cache pool is for caches of a source of truth; datasets without a fill path go to a serving store with bulk-load semantics. Puts a price on each tier per GB so teams can make the tradeoff themselves.

Key Takeaway: "If there's no source to fill from on a miss, it isn't a cache — it's a database with a cache's durability. Put it in a store designed for bulk-loaded serving."

What clears the Staff bar:

• Identifies that the request is a serving store, not a cache, from the missing fill path
• Uses access distribution to split RAM and SSD tiers with cost
• Protects existing tenants from bulk-load churn

Deep Dive 4: Post-Mortem — The Rolling Upgrade That Took Down Checkout#

Context: A routine minor-version upgrade of the session and cart cache pool (18 nodes, primary/replica pairs) caused a 25-minute checkout outage. The runbook said: upgrade replicas, fail over, upgrade old primaries. You are running the post-mortem.

Questions to Surface First:

• Were replicas fully synchronized before failover? How was that checked?
• How did clients discover the new primaries — and how long did they keep writing to the old ones?
• What happened to writes acknowledged by old primaries but not replicated?
• Why did a cart cache problem become a checkout outage — is the cart stored only in the cache?

Typical L5 Approach: Adds a check for replication offset before failover and slows the upgrade. Necessary; it misses that carts lived only in the cache.

Staff Approach: Timeline shows three compounding issues. Failovers happened while replication lag was ~4 seconds under peak load, so acknowledged cart writes were lost (asynchronous replication). Client topology refresh took up to 30 seconds, so some pods wrote to demoted nodes. And the cart service used write-behind: carts were written to cache and flushed to the DB every 60 seconds — so lost cache writes were lost carts, and checkout failed validation against stale carts. Fixes: carts written to the database first (cache-aside), the cache is a cache again; upgrade tooling waits for replica offset parity and runs off-peak; clients refresh topology on MOVED/connection errors immediately.

Principal Approach: Names the class: a cache silently promoted to a source of truth. Introduces a namespace registry field — source_of_truth: external | cache — where cache requires a durability review and is disallowed for money-adjacent data. Upgrade tooling refuses to proceed on pools whose namespaces declare themselves as source of truth until a review signs off.

Key Takeaway: "Write-behind turns a cache into a database with asynchronous replication and no backups. If you choose that, choose it in writing."

What clears the Staff bar:

• Finds the write-behind design as the root cause beneath the upgrade trigger
• Explains asynchronous replication loss on failover precisely
• Adds a durability declaration so the class of failure is visible before the next upgrade

Deep Dive 5: Multi-Region Expansion — Stale Fills From the Lagging Replica#

Context: After launching an EU region (reads local, writes forwarded to the US primary, DB replication ~300ms–3s), EU users report that edits "revert" — they save a change, see it, then see the old value for minutes. US users don't see it. Invalidations are broadcast to both regions' caches as soon as the US primary commits.

Questions to Surface First:

• When the EU cache entry is deleted, what does the next EU reader read — the EU replica? Has it applied the write yet?
• How does the author see their change initially — a read-your-writes bypass, or the response of the write itself?
• What is EU replica lag at p99 during peak?
• Are invalidations and replication on independent paths with independent delays?

Typical L5 Approach: Adds a short TTL in the EU region. Staleness drops to the TTL, but EU DB load triples and the bug persists within the TTL window.

Staff Approach: Diagnoses the cross-region stale fill: the invalidation (fast, ~100ms) beats replication (~1–3s); the next EU reader misses, reads the EU replica that doesn't yet have the write, and fills the old value, which then lives for the TTL. Fix: drive EU invalidations from the EU replica's own change stream, so the delete happens only after the local replica has the write. Interim: a second delete in the EU 5 seconds after the first (above p99 replica lag). For authors, a per-user, per-entity bypass marker for 10 seconds after their write.

Principal Approach: Turns it into a standard for multi-region caching: invalidation is always derived from the local replica's change stream, never from a remote commit. Includes it in the multi-region readiness checklist, and plans for per-entity home regions as writes become regional.

Key Takeaway: "In multi-region, an invalidation that outruns replication is worse than no invalidation — it invites a stale fill. Invalidate from the local replica's change stream."

What clears the Staff bar:

• Explains the race between invalidation and replication across regions
• Uses local CDC for ordering instead of shorter TTLs
• Handles read-your-writes for the author explicitly

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Identify whether a cache is a latency, load or compute shield, and derive the hit-rate floor from DB capacity
• Convert hit-rate changes into DB load and headroom in your head
• Write per-class staleness budgets and pick an invalidation strategy for each
• Draw the delete/fill race and close it with leases, versioned sets or ordered invalidation
• Design invalidation from the change log for multi-writer data, with lag SLOs and DLQs
• Protect the miss path: coalescing, fill budgets, stale-on-error, spare capacity, warm handoffs
• Handle hot keys with near caches and replication, with the bandwidth arithmetic
• Size a cluster from working set, throughput and network, and estimate its cost
• Operate a shared cache platform with isolation by placement, and explain why prefixes are not quotas
• Extend caching to a second region without cross-region stale fills

The Bar for This Question#

Mid-level (L4): Puts Redis in front of the database with cache-aside and a TTL; deletes on write. Knows LRU. Works in steady state; hasn't considered races, cold starts or hot keys.

Senior (L5): Adds consistent hashing, replicas, TTL jitter, maybe request coalescing; knows the thundering-herd problem. The gap: doesn't compute the DB's dependency on hit rate; staleness is "eventually consistent"; invalidation relies on every writer remembering; treats a cache outage as "slower". The design is sound until the first upgrade, launch or batch job.

Staff+ (L6): Starts from the cache's job and the hit-rate floor. States staleness in seconds per class with a product owner. Closes the delete/fill race with leases or versions and invalidates from the change log. Protects the database with fill budgets and a degraded path. Handles hot keys with near caches and bandwidth math. Separates load-bearing tenants. Measures staleness instead of assuming it. Names who pays: users for staleness, the DB team for misses, other tenants for shared-cluster churn. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "A Cache You Can't Restart Is a Database You Forgot to Back Up"#

The Staff position: If restarting the cache is dangerous, either engineer it as a capacity tier — replicas, warm handoffs, fill budgets, game days — or make the DB survive without it. Leaving it in between is the most common caching risk in production.

Why this matters in interviews: Asking "can we restart this cache at noon?" is a fast way to show you think about the cache as part of the system's capacity, not a bolt-on.

10.2 "Most Invalidation Bugs Are Ordering Bugs, Not Logic Bugs"#

The Staff position: The question isn't "did we delete?" but "did the delete happen after every read that could re-populate the old value?" Leases, versions and change-log ordering answer that; more deletes and shorter TTLs don't.

Why this matters in interviews: Candidates who frame invalidation as an ordering problem can reason through any variant the interviewer invents.

10.3 "Near Caches Fix More Hot-Key Incidents Than Resharding Ever Will"#

The Staff position: Popularity is concentrated by nature. Moving the hottest bytes into the process that needs them — for a bounded few seconds — beats any change to the cluster.

Why this matters in interviews: "Consistent hashing" is the reflexive answer to hot keys; correcting it with arithmetic is memorable.

10.4 "Shared Cache Clusters Are a Tragedy of the Commons"#

The Staff position: Share operations, not memory. A platform team should run the pools and the client; load-bearing tenants get isolated instances; everyone gets per-namespace dashboards and a cost per GB.

Why this matters in interviews: Showing you've thought about the cache as a multi-tenant platform is a strong L6-to-L7 bridge.

10.5 "Hit Rate Is a Vanity Metric Until You Weight It by Miss Cost"#

The Staff position: Report hit rate per namespace and weight misses by their cost: misses/s × cost per miss = DB seconds per second. That's the number the database feels, and it points to which namespace to optimize.

Why this matters in interviews: Moving from "our hit rate is 96%" to "our misses cost the DB 1.8 CPU-seconds per second, 70% from facets" shows you measure what matters.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

The Staff engineer builds a correct, resilient cache for one service. The Principal engineer counts the organization's caches and finds 37 Redis and Memcached clusters run by 22 teams, with four client libraries, three different ideas of what a TTL protects, and no inventory of which databases have been right-sized to which hit rates. Each cluster is reasonable; together they are a hidden capacity dependency of unknown size. The L7 problem is turning caching from a per-team performance trick into a governed capacity tier: one client that makes the safe protocol the default, declared staleness contracts, declared load-bearing status, and a platform that can upgrade, resize and fail over clusters without a war room.

The Org-Level Fault Line#

A caching platform vs caches owned by each service team.

🧭 Principal Move: "The platform owns the servers, the client library and the upgrade tooling. Teams own their namespaces — keys, staleness budgets, invalidation mappings — declared in a registry. Load-bearing namespaces get isolated pools and a hit-rate SLO the DB team signs. Prefix-sharing in one big cluster is what we're migrating away from."

Cost Model#

Assumptions: managed in-memory cache at ~$8–12 per GB-month including replicas' share; ~$250K fully loaded per engineer-year; DB read capacity priced as replica instances that would be needed to serve the traffic without the cache.

The pricing insight: the cache usually pays for itself many times over in DB capacity — which is exactly why its failure is expensive. The platform's budget should be justified against incident cost and DB headroom, not against the RAM bill. A 3-engineer investment in warm handoffs, fill budgets and per-namespace isolation is cheaper than one hour-long outage of a revenue path.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-CACHE-001: Caching Standard
Status: Approved   Owners: Caching Platform + Database Reliability + Security

Scope
  Every remote cache (and every near cache holding user-visible mutable data)
  used by production services.

MUST
  1. Register each namespace with: owner, data source, staleness budget,
     invalidation mechanism, load-bearing (yes/no), source_of_truth
     (external | cache).
  2. Use the platform cache client (leases on fill, miss coalescing, timeouts,
     breaker, per-pod fill budgets).
  3. Include a schema version in every key.
  4. Load-bearing namespaces: run on isolated pools with cross-AZ replicas, a
     declared hit-rate floor, and a DB fill budget agreed with the DB owner.
  5. Data with more than one writer, or any out-of-band write path, invalidates
     from the source's change stream.
  6. Caches of authorization state declare a revocation SLO approved by Security.
  7. source_of_truth = cache requires a durability review and is prohibited for
     payments, carts and authorization.

SHOULD
  1. Apply TTL jitter of ±10–20%.
  2. Enable sampled staleness checks (0.1% of hits) for namespaces with budgets
     under 60 seconds.
  3. Keep values under 100 KB; never exceed 512 KB.

Exceptions
  Filed with Caching Platform; reviewed within 5 business days; time-boxed to
  2 quarters. Security must approve exceptions to MUST 6.

Success metrics
  - Outages attributed to cold caches or cache maintenance: 0 per half-year
  - Namespaces with measured staleness within budget: ≥ 99%
  - Unregistered caches in production: 0 by end of year 2
  - Cache maintenance during business hours without incident: routine
  - Fleet RAM utilization: ≥ 60% (from ~30%)

What I'd Tell the VP#

"Our caches carry most of our read traffic — by my estimate, several of our databases would fall over within minutes if certain caches emptied, and nobody has a list of which ones. We've had two outages this year from routine cache maintenance and one security finding from a stale permission cache. I'm proposing a caching platform: one client library that makes the safe patterns automatic, a registry of what each cache holds and how stale it may be, and isolated pools for the caches our databases depend on. It's about five engineers for a year. It removes a recurring outage class, lets us do maintenance in daylight, and should cut cache spend by consolidating 37 clusters running at 30% utilization."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll engineer this cache with replicas, leases and near caching" — right for one service; ignores the 36 other clusters with none of it.
• "Hit rate is our SLO" — correct; missing that the DB owner should sign the floor and the fill budget.
• "We'll use CDC for invalidation" — right mechanism; missing who owns the pipeline's lag SLO and the per-table key mapping.

Appendices

Appendix A: Invalidation Mechanics in Depth#

A.1 Leases (Fill Tokens)#

def get(ns, key, loader):
    v, token = cache.get_or_lease(key)          # server returns value, or a lease token
    if v is not None:
        return v
    if token is None:                            # someone else holds the lease
        sleep(10ms); return get_retry(ns, key, loader, max_wait=200ms)  # or serve stale
    value, version = loader(key)
    cache.set_with_lease(key, value, version, token, ttl=jitter(ns.ttl))
    # rejected if a delete happened after the lease was issued
    return value

def invalidate(key):
    cache.delete(key)                            # also invalidates any outstanding lease

Two effects with one mechanism: only one caller per key fills at a time (the rest wait briefly or get a stale copy), and a fill based on data read before the delete can't land after it. Redis doesn't ship leases natively; implement with a small script: on miss, SET lease:{key} token NX PX 2000; on set, compare-and-set only if lease:{key} still equals the token; on delete, delete both.

A.2 Versioned Sets#

Store the source row's version (or commit LSN) with the value. A set succeeds only if its version is ≥ the version already cached — and deletes leave a tombstone with the deleting version for a few seconds so an older fill can't recreate the key.

SET_IF_NEWER(key, value, version):
    current = HGET key version  (or tombstone version)
    if current is None or version > current:
        HSET key value version; EXPIRE key ttl

A.3 Delayed Double Delete#

Delete on write, then delete again after a delay longer than p99 replica lag plus fill time (e.g., 1–2s in one region, 5s cross-region). Cheap, needs no server-side support, and narrows — but doesn't close — the race window. A reasonable interim fix; not the end state.

A.4 Read-Your-Writes for the Author#

After a user writes entity E, set a short-lived marker ryw:{user}:{E} (TTL ~2× replica lag, e.g., 10s). Reads by that user for E bypass the cache and read the primary. Everyone else sees the cache, within the staleness budget.

A.5 Miss Coalescing and Fill Budgets#

Appendix B: Keys and Data Model#

Values. Store the source version with the value. Compress values over ~1 KB (2–4× is typical for JSON). Avoid giant composite values for hot keys — split the frequently-read summary from the rarely-read detail.

Negative caching. Cache "not found" with a short TTL (e.g., 30–60s) to stop repeated lookups for missing IDs — including enumeration attacks — from reaching the DB. Invalidate negatives on create.

Appendix C: Coordination Mechanisms#

C.1 Warm Handoff for Planned Node Replacement#

C.2 Cross-Region Invalidation Ordering#

Driving each region's invalidations from its own replica guarantees the delete happens after the local data is new, so a refill can't pick up the old row.

C.3 Quick Comparison#

Appendix D: API Contract & Client Behavior#

Appendix E: Observability#

E.1 Core Metrics#

E.2 Control Plane vs Data Plane#

The data plane — client library, cache servers — must serve reads with the control plane down: clients cache topology and namespace config locally and keep working with the last-good version. The control plane — registry, upgrade tooling, CDC workers, warm-handoff orchestration — can be unavailable for minutes. Its failure stops changes and delays invalidations (raising staleness), but must never stop reads. Invalidation lag during a control-plane outage is the measurable cost; sampled staleness shows it.

E.3 Debugging the Silent Failure#

"Users see stale data sometimes" is the hardest cache ticket. The checklist: (1) sampled staleness for that namespace — is it over budget, and since when? (2) invalidation lag per partition — a stalled partition explains 1/N of entities; (3) write paths — any new writer bypassing the change stream? (4) regional skew — does it happen only in a replica region? (5) TTLs — did someone raise a TTL relying on invalidation that doesn't exist? Each step is a dashboard query when the metrics above exist, and a day of guesswork when they don't.

Appendix F: Scale Evolution#

What You Don't Build on Day One:

• CDC invalidation — until there's a second writer or an out-of-band write path
• A proxy tier — client-side routing is enough until connection counts or multi-language clients force it
• Cross-region cache replication — regional caches with local invalidation are simpler
• SSD-backed tiers — until the working set's RAM cost passes ~$20K/month
• A custom cache server — the leverage is in the client and the operating model

Appendix G: Multi-Tenancy, Fairness & Cost#

Isolation by placement. Memcached and Redis evict across the whole process; namespaces sharing an instance share fate. Real isolation means separate instances (or separate processes) per tenant group. A pool can contain many small instances and pack tenants onto them by size and criticality.

Chargeback. Charge namespaces per GB-month of reserved memory, and show them their hit rate and DB cost avoided next to it. Teams caching low-reuse data see a high cost with little avoided DB load and remove it themselves — the most effective cost optimization is usually deletion.

Value economics. Compression cuts memory 2–4× on text formats for ~10–50µs of CPU per value; for large values it's almost always worth it. Trimming fields nobody reads from cached objects is the next cheapest win.

Tiering. For compute-shield and long-tail data, SSD-backed caches (flash-backed Memcached variants, or key-value stores on NVMe) serve sub-millisecond to low-millisecond reads at a fraction of RAM cost. Keep the hot set in RAM and the long tail on flash.

Fairness under scarcity. When a pool is short of memory, evict from namespaces that exceed their reservation first. When the DB is short of capacity, fill budgets should be per namespace, with load-bearing, user-facing namespaces getting priority over background ones.