Memcached

Why This Matters#

Memcached is not a weaker Redis. It is a deliberate decision to keep the cache server dumb and put every distributed-systems problem in the client. The server stores bytes under keys in RAM, evicts the least recently used ones when full, and knows nothing about other servers. There is no replication, no persistence, no cluster membership, no failover. Which server owns a key, what happens when that server dies, how the cache stays consistent with the database and how a hot key's expiry is kept from flattening the database are all answered by the client library or a routing tier in front of the servers.

That is why "add a Memcached tier" is a sentence interviewers push on. The L5 candidate draws a cache box and says "99% hit rate." The L6 candidate says "a pool of 40 Memcached nodes behind a routing tier with consistent hashing; reads are look-aside with leases so only one client refills a missing key; writes update the database and then delete the key, never set it; when a node dies its traffic goes to a small gutter pool with short TTLs instead of being rehashed onto healthy nodes; and I alarm on the database's read QPS, because a 1-point drop in hit rate from 99% to 98% doubles database load." The L7 candidate asks whether the company needs a shared cache platform with routing, warmup and shadowing, who owns its capacity, and what a cold restart of the whole tier costs the databases behind it.

The L5 → L6 gap is not knowing what a slab is. It is knowing that the database behind the cache is provisioned for the cache's miss rate, so every cache failure mode is really a database failure mode with a delay.

The L5 → L6 → L7 Contrast#

The 60-Second Pitch#

"I'd put a Memcached pool in front of the user and product databases as a look-aside cache. It's multi-threaded, so one large node uses all its cores and we don't run a shard per core. The client or a routing proxy places keys with consistent hashing, so adding a node moves about 1/N of the keys. Reads use leases, or the meta protocol's win flag, so on a miss exactly one caller refills the key and others wait a few milliseconds or take the stale value. Writes go to the database first and then delete the key. Nothing in Memcached is the source of truth: no replication, no persistence, so a node restart is a cold node. I size the database for losing one node at peak and use a small gutter pool so a dead node's keys don't stampede the database. If we need sorted sets, counters with durability or a lock, that's Redis, not this."

The Three Intents#

🎯 Staff Move: "I'll design the first intent: a look-aside cache in front of the database for reads that can tolerate a few seconds of staleness. Everything in Memcached can be recomputed, and I'll size the database so losing a cache node at peak is a latency blip, not an outage."

The Staff Positions#

Architecture & Internals#

Only five internals change design decisions: the threading model, the slab allocator and LRU, client-side distribution, the protocol's concurrency primitives, and the absence of replication. The head-to-head is in Redis vs Memcached.

One Process, Many Threads, No Peers#

A Memcached server is a single process with a pool of worker threads (4 by default) handling connections via an event loop, listening on TCP 11211 with a default cap of 1,024 connections; UDP is off by default since 1.5.6 (configuration docs). Each server is independent: it never talks to another Memcached server.

Why it matters in design: because all cores serve one cache, a single 64-core node can do on the order of a million simple gets per second (typical, not a guarantee), and you size the fleet by memory and network, not by core count. Because there are no peers, there is no cluster to partition, no failover election and no replication lag. There is also nothing that notices a node is gone except the clients.

The Slab Allocator: Why a 600-Byte Item Can Cost 700 Bytes#

Memcached does not malloc per item. Memory given by -m is carved into 1 MB pages; each page is assigned to a slab class, and each class cuts its pages into fixed-size chunks. Chunk sizes grow geometrically by the growth factor (1.25 by default, -f), so classes look like 96, 120, 152, 192 … bytes up to the 1 MB maximum item size (-I) (UserInternals wiki). An item goes into the smallest chunk that fits key + value + ~50 bytes of header.

Three design consequences:

1. Internal fragmentation. A 200-byte item in a 240-byte chunk wastes 17%. With a 1.25 factor the worst-case waste per item is ~20%; Facebook's paper describes using a finer factor of 1.07 for their workload.
2. Eviction is per class. LRU runs inside each slab class, so a class that is short of pages evicts its items even while another class holds cold data. Since 1.5.0 the "modern" defaults are on: segmented LRU (HOT/WARM/COLD), an LRU crawler that reclaims expired items, and automatic slab rebalancing that moves pages between classes as sizes shift (1.5.0 release notes).
3. Value size shape matters. A workload that changes its value sizes (a new serialisation format, a bigger JSON payload) shifts demand between classes; until pages rebalance, hit rate drops on the classes now in demand. Watch evictions per slab class after any payload change.

Client-Side Distribution: The Ring Lives in the Client#

The server has no idea which keys it should own. The client library (or the router) hashes the key onto a consistent-hash ring with many virtual points per server (the classic "ketama" scheme), so adding or removing one of N servers moves roughly 1/N of the keys rather than nearly all of them.

modulo:      server = hash(key) % N
             N: 20 -> 21  => ~95% of keys change server => cache is effectively cold
consistent:  server = first point clockwise from hash(key) on a ring (100-200 vnodes/server)
             N: 20 -> 21  => ~1/21 = ~5% of keys move

The trap: every client must agree on the ring. Two services with different library versions, hash functions or server lists send the same key to different nodes: double memory, and invalidations that miss the copy the other service reads. That is the real argument for a routing tier: one place that owns the server list and the hash. Facebook built mcrouter, an open-source Memcached-protocol router that does consistent hashing, prefix routing, replicated pools, failover, shadowing and cold-cache warmup, and reported it handling close to 5 billion requests per second at peak (Meta engineering). Memcached itself now ships a built-in proxy (since 1.6.23), configured in Lua, that fronts pools of backend servers (proxy docs). The ring itself is covered in Consistent Hashing.

Concurrency Primitives: CAS, Leases and the Meta Protocol#

Facebook's paper reports that, for a set of keys especially prone to thundering herds, leases cut the peak database query rate from 17K/s to 1.3K/s (NSDI 2013 paper). The meta protocol brings the same idea to stock Memcached: the first client to miss gets W and refills; concurrent clients see Z and either wait briefly or serve the stale value marked with X.

🎯 Staff Insight: "Memcached's interesting features are all about the miss, not the hit. Hits are easy. What protects the database is that on a miss exactly one caller refills, and a write can't be overwritten by a slower reader holding an older value."

No Replication, No Persistence — On Purpose#

A restart is an empty node. A dead node's keys are gone. extstore can spill values to flash to extend capacity, but it is still a cache. Availability is something you build around the servers:

Data Modeling / Core Usage — "The Entire Game": The Miss Path#

With Redis the game is picking the data structure. With Memcached there is one structure, so the game is the miss path: what happens on a miss, on a write and on an expiry. Get these right and the database never notices the cache's imperfections; get them wrong and the cache becomes the reason the database falls over.

The miss-path contract (per key family):
  1. key      -> namespaced, versioned, <= 250 bytes, one owner
  2. read     -> look-aside; on miss exactly one refiller (lease / W flag)
  3. write    -> DB first, then DELETE the key (never set from the writer)
  4. expiry   -> TTL as a backstop, with jitter; early recache for hot keys
  5. batch    -> multi-get per server; bound fan-out per request

Step 1: Key Design#

key = <service>:<entity>:v<schema_version>:<id>[:<variant>]
      user-svc:profile:v3:81723
      feed-svc:timeline:v7:81723:page1

rules:
  - keys <= 250 bytes, no spaces or control characters (protocol limit)
  - schema version in the key -> a format change is a deploy, not a flush
  - one service owns a key prefix; routers can send prefixes to different pools
  - never embed unbounded input (search strings) without hashing it

Bumping v3 to v4 is a mass invalidation without a flush: old keys age out by TTL and LRU. Plan for the cold-miss load that follows: a version bump on a hot family is, to the database, the same as losing that slice of cache.

Step 2: The Read Path With Herd Protection#

value, flags = mg(key, v, N30, R10, t)      # meta get: vivify on miss, recache if TTL < 10s
if flags has W:                              # we won: we refill
    value = db.read(id)
    ms(key, value, T=ttl_with_jitter())      # set; clears the win marker
elif flags has Z and value is stale (X):     # someone else refilling; serve stale
    return value
elif flags has Z and no value:               # someone else refilling; brief wait
    sleep(5-20 ms); retry once; else read DB with a concurrency cap
return value

On a hot key that expires while serving 20,000 requests per second, the naive path sends every request in the refill window (say 50ms) to the database: ~1,000 identical queries. With a win flag or lease, it sends one. The wider read-path design is in Read-Heavy Systems.

Step 3: The Write Path — Delete, Don't Set#

Without the lease, the reader's late set of version 1 lands after the writer's delete and the cache holds stale data until the TTL expires. With "set on write" instead of delete, two writers racing can leave either value cached. Delete-plus-lease makes the database the only arbiter. For data that must be invalidated across regions or by many writers, Facebook's paper describes tailing the database's commit log and issuing deletes from there (mcsqueal), so invalidation is driven by what actually committed. The trade-offs among these consistency choices are in Caching Basics.

Step 4: TTLs and Jitter#

Jitter matters because a batch job that warms 2 million keys with the same TTL schedules 2 million simultaneous misses. ttl × uniform(0.9, 1.1) spreads them over minutes.

Step 5: Multi-Get and the Fan-Out Problem#

A page that needs 200 keys spread across 50 servers issues ~50 parallel requests, and its latency is the slowest of the 50. Two effects matter at scale:

• Tail amplification. If each server's p99 is 1ms, a 50-way fan-out sees at least one p99-or-worse response on ~40% of requests. Batch by server, cap fan-out, and put the most critical keys in a small, replicated pool.
• Incast. Fifty responses arriving at one client at once can overflow switch buffers. Facebook's paper describes a client-side sliding window that limits outstanding requests, growing on success and shrinking on timeouts.

The Latency Budget calculator is the fastest way to show this in an interview.

🎯 Staff Move: "Reads are look-aside with a win flag so one caller refills; writes update the database and delete the key; TTLs are jittered backstops. If they ask about consistency, I'll say the cache is eventually consistent with a bound set by the TTL, and leases stop the one race that would make it permanently stale."

The Tunable Tradeoff — Hit Rate × Memory × Staleness#

database_read_load = cache_request_rate x (1 - hit_rate)

  1M req/s at 99.0% hit -> 10K req/s to the DB
  1M req/s at 98.0% hit -> 20K req/s to the DB   (1 point of hit rate = 2x DB load)
  1M req/s at 99.5% hit ->  5K req/s to the DB

Hit rate vs memory follows the key-popularity curve: the first GB buys the most;
each additional GB buys less. Measure it (shadow a bigger pool, or replay traces).

🎯 Staff Move: "I'll treat hit rate as a database capacity number. At 1M reads per second, 99% versus 98% is 10K versus 20K database reads, so I'd rather pay for 20% more cache memory than a bigger database, and I'll replicate only the hot pool where a miss is expensive."

Anti-Patterns — What Kills Memcached Deployments#

1. Memcached as the Only Copy#

Sessions, shopping carts or rate-limit state stored only in Memcached vanish on restart, eviction or a node replacement, and the eviction is silent. Fix: anything that must survive goes to a database or a replicated store such as Redis with persistence; Memcached caches it at most.

2. Modulo Hashing#

hash(key) % N works until the fleet changes size. Adding one node to 20 remaps ~95% of keys, which is a full cold start of the cache and a database overload. Fix: consistent hashing in every client, ideally via one shared library or a router.

3. No Herd Protection on Hot Keys#

A celebrity profile, the home-page config, a flash-sale product: each expiry sends thousands of concurrent misses to the database. Fix: leases or meta-protocol win flags, early recache (R flag) before expiry, and a near cache for the top keys. See Hot Keys.

4. Rehashing a Dead Node's Keys Onto Live Nodes#

Automatic ejection plus rehash looks self-healing. In practice the node that inherits a hot key overloads, gets ejected too, and the failure walks around the ring. Fix: gutter pool, or replicated pools for hot data; eject slowly and deliberately.

5. Set-on-Write Invalidation#

Writers updating the cache directly race with readers refilling it; one ordering leaves stale data until TTL. Fix: delete on write, leases on refill, TTL backstop.

6. Huge Values and Wide Fan-Out#

A 900 KB serialized object read on every request saturates a node's NIC long before its CPU; a page that fetches 500 keys across 100 servers lives at the fleet's p99. Fix: split large values, compress, cache smaller projections; bound keys per request; batch by server.

7. Rolling Restarts Without Warmup#

Upgrading a 100-node pool one node every minute empties 1% of the cache per minute; the whole pool is cold within two hours, during business hours. Fix: restart slowly with hit-rate gates, use a router's warmup (cold node reads through from a warm replica), or shift traffic to a warm pool first.

8. One Shared Pool for Every Team#

A batch job that caches 200 GB of one-off values evicts every other team's hot set. Fix: pools per workload behind a router's prefix routing, each with its own memory budget and hit-rate SLO.

The Technology Landscape — Head-to-Head Comparison#

🎯 Staff Insight: "For a pure look-aside cache either works, so the tiebreaker is what's coming next. If product will ask for counters, leaderboards or locks, that's Redis. If the job is a huge, boring get/set tier where cost per GB and cores per node are the bill, Memcached's dumbness is the feature."

Patterns#

Pattern 1: Look-Aside With Leases#

The default. Application reads cache, on miss reads the database and refills under a lease or win flag; writes go to the database and delete. Works for 90% of read-heavy data. See Distributed Cache for the full design.

Pattern 2: Routing Tier With Prefix Pools#

One router owns the server lists, the hash and the failover policy. Teams get isolated pools by key prefix with separate budgets. Shadowing lets you test a new instance type with real traffic before moving any keys.

Pattern 3: Gutter Pool for Failures#

A small pool (~1% of servers in Facebook's account) that takes over failed servers' gets with short TTLs. A failed get goes to the gutter; on miss the client reads the database and fills the gutter. Hot keys spread across idle gutter servers instead of piling onto one live node.

Pattern 4: Zone-Replicated Cache#

Write to a copy in every availability zone; read from the local zone; on a miss, fall back to another zone before the database. Reads stay in-zone (lower latency, no cross-zone transfer cost) and a zone loss doesn't empty the cache. This is the model Netflix's EVCache wiki describes.

Pattern 5: Commit-Log Invalidation#

Instead of every application server issuing deletes after writes, a daemon tails the database's replication or commit log and deletes the affected keys, batching them through routers. Invalidation follows what committed, survives application crashes, and replays after an outage. The same idea in event form is Transactional Outbox.

Scaling#

The Numbers#

Sizing a Pool#

Example: product-detail cache (illustrative)
  working set: 80M products x 2 KB avg value          = ~160 GB of values
  with header, key and ~15% slab waste                = ~190 GB to hold everything
  hot 30% of products serve ~95% of reads             -> ~60 GB holds the hot set
  provision 120 GB for the warm tail                  -> measured hit rate 99.2%

  traffic: 400K reads/s -> normal misses 0.8% = 3.2K reads/s to the database

  option A: 6 nodes x 20 GB    lose 1 -> ~1/6 of keys miss -> ~+66K reads/s
  option B: 40 nodes x 3 GB    lose 1 -> ~1/40 of keys miss -> ~+10K reads/s

Same memory, very different failure. Losing one node of six puts ~17% of keys on the database at once, a ~20× jump over normal misses; losing one of forty is ~4×. The answers are more, smaller nodes (losing 1 of 40 is 2.5%), a gutter pool, or replication for the hot pool. Node count is a blast-radius decision as much as a memory one. The Sharding Planner and Cost Estimator help frame it.

Scaling Moves in Order#

1. Fix the miss path first: herd protection, delete-on-write, jittered TTLs. Often worth more than any memory you could add.
2. Add memory until the hit-rate curve flattens, measured by shadowing or trace replay, not guessed.
3. Split pools by workload so a low-value family can't evict a high-value one.
4. Add a routing tier once more than a handful of services share the fleet: one ring, connection pooling, failover, shadowing.
5. Replicate hot pools (per zone) where a miss is expensive; leave the long tail unreplicated.
6. Add regional pools and commit-log invalidation when going multi-region, so each region reads locally and writes invalidate everywhere.

🎯 Staff Move: "I'd rather run 40 smaller nodes than 6 big ones. Memory is the same, but losing a node becomes a 2.5% miss event instead of 17%, which is the difference between the database noticing and the database falling over."

Failure Modes & Recovery#

1. Node Loss → Miss Storm#

• Symptom: One cache node dies; database read QPS jumps several-fold within seconds; p99 latency across services climbs.
• Root cause: That node's 1/N of keys all miss at once, and the database was provisioned for the healthy miss rate.
• Detection: Per-node get_hits dropping to zero; database read QPS vs baseline; client connection errors to one host.
• Fix: Route failed gets to a gutter pool; shed or degrade non-critical reads; replace the node.
• Prevention: More, smaller nodes; gutter pool; replication for hot pools; database headroom sized for one node loss at peak.

2. Thundering Herd on a Hot Key#

• Symptom: Periodic database spikes aligned with a popular key's TTL; identical queries arriving by the hundred.
• Root cause: Expiry or delete of a hot key with no refill coordination.
• Detection: Database slow-query logs showing identical statements in bursts; per-key miss rate from client-side sampling.
• Fix: Enable leases or meta win flags; serve stale while one caller refills; early recache.
• Prevention: Herd protection in the shared client library by default; hot-key detection.

3. Permanently Stale Values (the Stale-Set Race)#

• Symptom: A user changes their name; some pages show the old name for an hour, then it fixes itself.
• Root cause: A reader's refill with an old value landed after the writer's delete; no lease to reject it, so it lived until TTL.
• Detection: Sampled cache-vs-database consistency checks; user reports correlated with write bursts.
• Fix: Delete the affected keys; shorten TTL temporarily.
• Prevention: Leases (or add plus versioned values); delete-on-write only; commit-log invalidation for multi-writer data.

4. Eviction of the Hot Set (Slab Imbalance or Noisy Pool)#

• Symptom: Hit rate drops from 99% to 95% after a deploy that changed value sizes, or after a batch job started caching.
• Root cause: Demand shifted to slab classes with too few pages, or a new workload is evicting others in a shared pool.
• Detection: evictions and evicted_unfetched per slab class; hit rate per key prefix; slabs_moved.
• Fix: Let automove rebalance (or force it); move the noisy family to its own pool; add memory.
• Prevention: Pools per workload; alarm on per-class evictions after payload changes.

5. Cold Cache After a Mass Restart#

• Symptom: After a deploy, kernel patch or zone recovery, hit rate starts near zero and the database is at its limit for an hour.
• Root cause: Memcached has no persistence; restarted nodes come back empty.
• Detection: Fleet-wide hit rate; number of nodes with uptime < 1 hour.
• Fix: Slow the restart; enable warmup from a warm replica or zone; throttle traffic to the database.
• Prevention: Restart pacing gated on hit-rate recovery; warm-pool cutovers; Facebook describes cold-cluster warmup bringing a cluster back to full capacity in a few hours instead of a few days.

Operational Reality Matrix#

When to Use vs. Alternatives#

When NOT to Use Memcached#

• The data has no other home. Sessions, carts, rate-limit counters you can't lose.
• You need anything beyond get/set/delete/incr/cas. Leaderboards, sets, streams, scripts: Redis.
• The fleet is small and the team already runs Redis. Two caching technologies for a handful of nodes is operational cost without benefit.
• Strong consistency with the database is required. A cache is eventually consistent by construction; reads that must see the latest write go to the primary.
• Values are huge and rarely reused. Caching 5 MB blobs read once a day is object storage's job.

Operational Concerns#

What the On-Call Actually Does#

1. Watches the database, not just the cache. A cache incident shows up first as database read QPS and latency.
2. Replaces dead nodes and watches the miss storm subside; confirms gutter or replica pools absorbed it.
3. Paces restarts and upgrades with hit-rate gates: next node only when the previous one is back above ~95% of its normal hit rate.
4. Investigates hit-rate drops: per-prefix hit rates, per-class evictions, a deploy that changed value sizes or bumped a key version.
5. Hunts hot keys from client-side sampling and router stats; moves them to a near cache or replicated pool.
6. Owns the router config: pool membership, prefix routes, failover thresholds, shadowing for new hardware.

Key Metrics & Alerts#

Interview Application — Staff-Level Plays#

Which Case Studies Use Memcached#

Every System Design Question Has a Memcached Moment#

• News feed: "Feed pages assemble ~100 objects; I batch multi-gets by server and keep the top 1,000 hot objects in a 2-second near cache, so a celebrity post doesn't melt one node."
• URL shortener: "Redirects are look-aside with a 24-hour jittered TTL and 60-second negative caching for unknown codes, so scanners hammering random codes don't reach the database."
• E-commerce: "Product pages cache with delete-on-write from the catalog service; inventory counts are never cached, because a stale 'in stock' is a broken promise."

What Interviewers Probe#

Common Interview Mistakes#

L5 vs L6 vs L7 Responses#

The Staff Memcached Checklist#

1. Miss budget: "The database absorbs 2× normal misses, so the hit rate floor is 99% and losing a node can't exceed that."
2. Distribution: "Consistent hashing in one shared client or router; adding a node moves about 1/N of keys."
3. Read path: "Look-aside with leases or meta win flags; stale-while-revalidate for hot keys."
4. Write path: "Database first, then delete; TTL with jitter as a backstop."
5. Failure: "Many small nodes, a gutter pool, replicated pools only for hot data."
6. Ownership: "Platform owns nodes and routing; teams own key prefixes, TTLs and invalidation."

🎯 Staff Insight: Don't use Memcached as a store of record, as a session store, for anything needing structures or atomic multi-step logic, or where reads must see the latest write. The strongest Memcached signal is doing the miss-storm arithmetic for one dead node out loud.

Evaluation Rubric#

Strong hire signals

Lean no-hire signals

Common false positives

• Slab internals ≠ cache design. Knowing chunk sizes says nothing about the miss path.
• "Our hit rate is 99.9%" ≠ resilience. Ask what the database sees when a node dies.

Beyond Staff: The Principal View#

Why L7 Sees This Problem Differently#

At Staff level Memcached is a well-configured cache in front of one database. At Principal level, the cache fleet is load-bearing infrastructure for every database in the company: the databases are sized for its hit rate, so its failure modes are the databases' failure modes, and its capacity is a shared budget that one team's batch job can spend for everyone. The L7 question is "Do we run caching as a platform with a router, pools, quotas and warmup, or as per-team fleets, and who owns the database headroom a cache loss implies?"

🧭 Principal Move: "Every database behind a cache is quietly provisioned for the cache's miss rate. So I want one number per database: the miss rate it can survive. Then cache capacity, replication and node size are set to keep the worst plausible cache incident under that number, and the database owner and the cache owner sign it together."

The Org-Level Fault Line#

Cache as a shared platform vs per-team cache fleets.

The Principal default: a platform-owned cache tier with a routing layer, per-team pools by key prefix, chargeback by provisioned memory, herd protection and delete-on-write in the shared client library, and a "survive one node at peak" standard negotiated with each database owner.

🧭 Principal Insight: "The router is where cache policy lives. Without one, every service is its own cache platform and every incident is a new discovery."

Cost Model#

Assumptions: memory-optimised node with ~50 GB usable cache at ~$500/month; loaded engineer ~$21K/month; savings expressed as database capacity the cache displaces. Directional only.

The lever is hit rate per dollar of memory: measure the hit-rate curve per pool and stop adding memory where it flattens. The hidden cost is database headroom for cache loss, often larger than the cache bill itself.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-CACHE-003: Look-Aside Cache Contract

Scope: Every cache pool fronting a production datastore.

MUST
  1. Cache only data recomputable from a named source of truth.
  2. Use the platform client or router (one hash ring, herd protection on).
  3. Invalidate by delete after the source write commits; never set from writers.
  4. Set a TTL with >= 10% jitter on every key family.
  5. Declare, with the database owner, the miss rate the database can survive,
     and keep single-node loss at peak below it.
SHOULD
  6. Use versioned key prefixes for format changes.
  7. Serve stale on refill for hot key families.
  8. Replicate pools whose misses cost > 10 ms or hit a constrained database.

Exceptions: platform review, time-boxed.
Success metrics: zero database incidents caused by a single cache node loss;
  per-pool hit-rate SLOs met; zero cache-only state.

What I'd Tell the VP#

"Our databases only stay up because the cache in front of them answers about 99 of every 100 reads. That means a cache problem becomes a database outage within seconds, and today each team runs its cache differently. I'm proposing one shared cache platform with standard safety features, and a simple rule agreed with every database owner: losing one cache server at peak must never take a database down. It costs about one engineer and some extra servers, and it removes a class of outages we've had twice this year."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "Add a gutter pool" without who sizes the database for the residual misses.
• "Use leases" as a per-service fix rather than a library default for every team.
• "Memcached for the cache, Redis for the rest" without asking whether running both is worth it at this fleet size.

How Real Companies Built It#

Facebook — Scaling Memcache#

Facebook's NSDI 2013 paper describes building a distributed key-value cache on Memcached that handles billions of requests per second and holds trillions of items. It introduces leases (a 64-bit token on miss; a delete invalidates it; tokens issued at most once every 10 seconds per key), which cut the peak database query rate for herd-prone keys from 17K/s to 1.3K/s; Gutter, about 1% of servers in a cluster that absorb a failed server's traffic with quickly expiring entries instead of rehashing onto live servers; invalidation daemons (mcsqueal) that tail the database commit log; and cold cluster warmup, which brings an empty cluster to full capacity in a few hours instead of a few days (USENIX NSDI '13).

Staff insight: Every mechanism in that paper is about the miss path and failure handling, not hits. Leases, gutter and commit-log invalidation are the three sentences that turn a cache box into a Staff answer.

Meta — mcrouter#

Facebook open-sourced mcrouter, a Memcached-protocol router that looks like a server to clients and a client to servers. It provides connection pooling, consistent hashing, prefix routing to separate pools, replicated pools with failover, traffic shadowing for testing new hardware, cold-cache warmup and online reconfiguration, and was handling close to 5 billion requests per second at peak; Instagram and Reddit were early external users (Meta engineering).

Staff insight: The router is where a dumb cache becomes a platform. In an interview, proposing a routing tier is the answer to "how do 50 services share a cache safely?"

Netflix — EVCache#

Netflix's EVCache is an open-source distributed in-memory store built on Memcached and the spymemcached client for AWS; the name stands for Ephemeral, Volatile Cache (EVCache on GitHub). Its documentation describes writing to every availability zone, reading from the local zone for latency, and optionally falling back to another zone when data is missing, which is much faster than going to the source (EVCache wiki).

Staff insight: Replication was added in the client, not the server, and paid for in memory per zone. That's the honest answer to "Memcached has no replication": you build exactly as much as the miss cost justifies.

Practice Drill#

Drill 1: The Cache Node That Took Down the Database#

Prompt: "Your product catalog runs 6 large Memcached nodes (64 GB each) in front of a PostgreSQL primary with 4 read replicas, at 400K reads per second and a 99.2% hit rate. Yesterday one cache node's host failed; the replicas hit 100% CPU, and the site was degraded for 25 minutes. The team proposes doubling the cache node size. What do you do?"

Quick Reference Card#

Model:           dumb, multi-threaded, volatile LRU cache; servers never talk to each other
Distribution:    client or router consistent hashing; ~1/N keys move per node change
                 (modulo moves ~(N-1)/N)
Defaults:        port 11211 TCP (UDP off since 1.5.6); 4 threads; 1,024 connections;
                 1 MB max item; 1 MB slab pages; growth factor 1.25; keys <= 250 bytes
Memory:          slab classes with fixed chunks; per-class segmented LRU; automove
                 rebalances pages (modern defaults since 1.5.0)
Durability:      none; restart = empty node; extstore extends to flash, still a cache
Herd control:    leases (token on miss, 1 per 10s per key at Facebook) or meta protocol
                 flags N / W / Z / R / X / I
Write path:      DB first, then DELETE; lease rejects stale sets; TTL + jitter backstop
Failure:         one of N nodes lost = 1/N of keys miss at once
                 -> many small nodes, gutter pool (~1% of servers), replicated hot pools
Hit rate math:   DB load = requests x (1 - hit rate); 99% -> 98% doubles DB reads
Routing tier:    mcrouter or the built-in proxy (1.6.23+): pools, prefix routing,
                 failover, shadowing, warmup

RED FLAGS
  - Memcached as the only copy of sessions, carts or counters
  - hash(key) % N
  - Set-on-write invalidation
  - No herd protection on hot keys
  - Rehashing a dead node's keys onto live nodes
  - A few huge nodes (big blast radius per failure)
  - Rolling restarts without hit-rate gates
  - One pool shared by batch jobs and the hot path