Design with Cassandra — Staff-Level Technology Guide

Why This Matters#

Cassandra is not a database you query. It is a database you pre-answer. You decide, at schema time, every question the system will ever be asked, and you lay the data on disk in exactly the order those answers will be read. Ask it a question you didn't plan for and it either refuses (ALLOW FILTERING required) or scans the cluster to find out. That's not a limitation to apologize for — it's the bargain that buys you linear write scaling, no single primary, and multi-datacenter writes that survive a region loss.

That bargain is why Cassandra keeps appearing in Staff loops. Messaging history, activity feeds, notification inboxes, IoT telemetry, fraud signals, user event timelines — any problem with high write volume, simple access patterns, and an availability requirement stronger than its consistency requirement has a Cassandra answer. The L5 candidate says "Cassandra, because it scales writes." The L6 candidate says "table notifications_by_user, partition key (user_id, month), clustering created_at DESC, RF=3 per DC, LOCAL_QUORUM both ways, TWCS with 30-day windows, TTL 90 days — and I've bounded the partition to ~2MB for a heavy user." The L7 candidate asks who will run repair for the next five years, because the most common way Cassandra deployments fail isn't load — it's operational neglect.

The L5 → L6 gap is not knowing what a memtable is. It is knowing that the partition key is the query, the partition size is the risk, and deletes are writes that come back to haunt you.

The L5 → L6 → L7 Contrast#

The 60-Second Pitch#

"I'd store the notification inbox in Cassandra. It's a write-heavy, append-mostly workload — 40K inserts per second at peak, reads are always 'latest N for this user' — which maps to one partition per user per month, sorted newest first, so every read is a single-partition slice in a few milliseconds. RF=3 in each of two regions with LOCAL_QUORUM gives us read-your-writes within a region and keeps writing if a whole region goes down. Data expires by TTL at 90 days with time-window compaction, so we never pay for tombstone scans. I would not use it for anything needing cross-row transactions or ad-hoc queries — the unread-count badge is fine as a counter, but billing stays in Postgres."

The Three Intents#

🎯 Staff Move: "I'm choosing Cassandra for the time-ordered intent: append-heavy, read the recent slice, expire by age. The one place I need a uniqueness guarantee I'll use an LWT on a tiny dedicated table — I won't let Paxos leak into the hot write path."

The Staff Positions#

Architecture & Internals#

Only the internals that change design decisions.

The Ring, Tokens, and Replica Placement#

Every row's partition key is hashed (Murmur3) to a 64-bit token. The token space is a ring divided among nodes; each node owns several token ranges via vnodes (num_tokens, default 16 since 4.0 — down from 256, which made repair and streaming painfully slow). A partition lives on the node owning its token plus the next RF−1 nodes clockwise that sit in different racks (with NetworkTopologyStrategy), so losing one rack or AZ never removes more than one replica.

There is no primary. Any node can coordinate any request. The client driver should be token-aware (send the request straight to a replica, saving a hop) and DC-aware (never cross regions for LOCAL_* levels).

The Write Path#

Writes never read and never update in place: append to the commit log, insert in memory, ack. That's why a single node can absorb tens of thousands of writes per second at sub-millisecond server time — and why every update and delete is just another write that compaction must later reconcile.

Design consequences:

• Inserts and updates are the same operation (upsert). There is no "insert fails if exists" without an LWT.
• Last write wins per cell, by client or coordinator timestamp. Clock skew between app servers decides conflicts — use server-side timestamps or a single writer per entity when order matters.
• Commit log sync mode (periodic every 10s by default) means a replica can lose up to ~10s of acked writes if that node crashes — which is why durability comes from RF=3 + quorum, not from fsync.

The Read Path#

A read must merge the memtable and every SSTable that might hold the partition. Bloom filters (per SSTable) skip files that definitely don't contain it; the partition index finds the offset in those that might.

Compaction — the Strategy Decides Your Disk and Your Reads#

🎯 Staff Insight: Picking TWCS for TTL'd time-series is one of the highest-leverage single decisions in a Cassandra design. It turns "delete 2 billion expired rows" into "unlink 30 files."

Tombstones, gc_grace_seconds, and Resurrection#

A delete writes a tombstone. Tombstones must live at least gc_grace_seconds (default 864000 = 10 days) so that a replica that was down during the delete learns about it via repair before the tombstone is purged. If a replica misses the delete and repair doesn't run within that window, compaction drops the tombstone on the other replicas — and the down replica's old value comes back to life on the next read repair.

The Three Anti-Entropy Mechanisms#

Data Modeling / Core Usage — "The Entire Game"#

In a relational database you model the entities and let the planner answer queries. In Cassandra you model the queries and accept that the same entity is written to several tables. Get the model right and the cluster is boring for years. Get it wrong and no amount of hardware fixes it — the only fix is a new table and a backfill.

Step 1: Write Down Every Read Query#

Worked example: a notification inbox for a 100M-user app.

Q4 is the trap. It has a different partition key (campaign) and is rare. Options: a second table written on every insert (write amplification for a rare query), or an offline job over a Spark/export snapshot. Rare, latency-tolerant queries belong outside the hot cluster.

Step 2: Partition Key = The Query's Equality Predicate (+ a Bound)#

Q1 is "by user, newest first." The naive key user_id puts a user's entire history in one partition — unbounded for heavy users and bots. Add a time bucket:

CREATE TABLE inbox.notifications_by_user (
    user_id     uuid,
    bucket      text,          -- 'YYYY-MM'
    created_at  timeuuid,
    notif_id    uuid,
    kind        text,
    actor_id    uuid,
    payload     text,          -- small JSON, < 2KB
    read        boolean,
    PRIMARY KEY ((user_id, bucket), created_at)
) WITH CLUSTERING ORDER BY (created_at DESC)
  AND default_time_to_live = 7776000          -- 90 days
  AND compaction = {'class': 'TimeWindowCompactionStrategy',
                    'compaction_window_unit': 'DAYS',
                    'compaction_window_size': 7}
  AND gc_grace_seconds = 172800;              -- 2 days: TTL'd, rarely deleted

Size the partition out loud:

heavy user:   200 notifications/day × 31 days = 6,200 rows/month
row size:     ~400 bytes on disk
partition:    6,200 × 400B ≈ 2.5MB   -> well under the 10MB target
pathological: a bot account at 10K/day = 310K rows, ~124MB  -> rate-limit at ingest

Q1 reads the current month's partition with LIMIT 50; if it returns fewer than 50 rows, the client reads the previous month. That's at most two single-partition reads.

Step 3: Clustering Columns = The Sort Order You Read In#

Clustering columns sort rows within a partition on disk. created_at DESC means "latest 50" is a sequential read from the start of the partition — no sort, no scan. You get one sort order per table; a second order means a second table.

Step 4: Denormalize, Then Own the Consistency Between Tables#

Q2 (unread count) can't be computed by scanning Q1's table at 120K/s. It gets its own table:

CREATE TABLE inbox.unread_counts (
    user_id uuid PRIMARY KEY,
    unread  counter
);

Now one logical event — "new notification" — writes two tables. They can diverge (a counter increment times out and is retried = double count; counters aren't idempotent). The Staff answer accepts that the badge is approximate and reconciles: when the user opens the inbox, the client recomputes unread from Q1's slice and the service resets the count. Name the owner of that drift.

Step 5: Choose the Alternate-Access Strategy Deliberately#

Primary-Key Pattern Cheat Sheet#

🎯 Staff Move: "Before I write any CREATE TABLE, here are the four reads and their rates. Three of them hit Cassandra as single-partition slices. The fourth is an ops query — I'm sending it to the analytics copy so it can't hurt the inbox."

The Tunable Tradeoff — Consistency Levels#

Cassandra lets each request choose how many replicas must respond. With replication factor RF, a read at level R and a write at level W are guaranteed to overlap on at least one replica when:

R + W > RF
RF=3: QUORUM (2) + QUORUM (2) = 4 > 3   -> overlapping, read sees the latest acked write
RF=3: ONE (1) + ONE (1)       = 2 ≤ 3   -> may read a replica that hasn't seen it
RF=3: ONE write + ALL read     = 4 > 3   -> consistent, but any one node down = reads fail

What Quorum Does Not Give You#

• Not linearizability. Two concurrent LOCAL_QUORUM writes to the same cell are resolved by timestamp, not order of arrival. If app servers' clocks differ by 50ms, the "later" write can lose.
• Not cross-partition atomicity. A logged batch eventually applies all mutations, but readers can see a partial batch in between.
• Not cross-DC read-your-writes. A LOCAL_QUORUM write in us-east is not guaranteed visible to a LOCAL_QUORUM read in eu-west until async replication lands (typically 100ms–1s; minutes during a partition).
• Not "failed means not written." A write that times out at the coordinator may still have been applied on some replicas and will propagate. Retries must be idempotent.

Lightweight Transactions (LWT)#

INSERT INTO accounts.usernames (username, user_id, created_at)
VALUES ('ada', 9f2c..., toTimestamp(now()))
IF NOT EXISTS;
-- [applied] = true  -> you own it
-- [applied] = false -> returns the current owner

LWT runs Paxos on the partition: linearizable for that partition, at ~4–10× the latency of a plain write and with sharply worse throughput under contention (competing proposers retry). Mixing LWT and non-LWT writes on the same cells breaks the guarantee.

Who Pays for Each Choice#

🎯 Staff Move: "LOCAL_QUORUM reads and writes, RF=3 per region. That gives read-your-writes within a region and keeps both regions writable if they partition. I'm accepting that a user who writes in one region and immediately reads from the other might not see it — sticky regional routing makes that rare."

Anti-Patterns — What Kills Cassandra Deployments#

1. Unbounded Partitions#

PRIMARY KEY (user_id, ts) with no bucket. Fine for two years, then a handful of power users and bots reach 1GB partitions. Reads time out, compaction of that partition stalls, repair streams the whole thing on every mismatch, and heap pressure causes GC pauses that hurt other tenants on the node. Fix requires a new table and a backfill. Prevention: a size bound in the design doc and an alert on nodetool tablehistograms max partition size > 100MB.

2. Queue-Shaped Workloads#

Insert job, worker reads oldest, deletes it. Every consumed job leaves a tombstone at the front of the partition — exactly where the next read starts. After a busy day, "read oldest" steps over 100K tombstones and fails. Cassandra is a terrible queue. Use Kafka, SQS, or a database with in-place deletes.

3. ALLOW FILTERING and Scatter-Gather Reads#

Any query without the full partition key must contact every token range. Latency grows with cluster size and one slow node sets the p99 for every such query. If it's in a request path, it's a missing table.

4. Read-Before-Write#

"Read the row, check a field, write the new value." It doubles latency, and two concurrent requests both read the old value — so it doesn't even work. Either make the write unconditional (upsert/LWW is fine), use an LWT for true conditions, or move the logic to a store with transactions.

5. Inserting Nulls#

Writing null to a column writes a cell tombstone. An ORM that sends every column on every insert can generate millions of tombstones a day without a single DELETE. Use unset values in prepared statements.

6. Large Multi-Partition Batches as a Bulk-Load Tool#

Logged batches spanning many partitions make one coordinator do all the work and write a batchlog. Batches are for atomicity across a few related writes, not throughput. For bulk loading, send many async single-partition writes or use SSTable loading.

7. Skipping Repair#

Everything works for months. Then a node that was down for 12 hours (hints expired after 3) returns, repair never ran within 10 days, tombstones are purged elsewhere — and deleted user data reappears. In a privacy context, that's a regulatory incident. Schedule repair with Reaper; alert on "time since last successful repair per table" > 7 days.

8. Oversized Nodes#

Packing 8–10TB onto a node to save money. Replacing it streams for a day or more, during which the cluster runs at reduced redundancy, and compaction can't keep up. Classic planning density is 1–2TB per node; 4.0+ zero-copy streaming and 5.0's UCS stretch that, but measure your replace time before going past ~4TB.

The Technology Landscape — Head-to-Head Comparison#

How to choose in one breath: sustained high write volume, multi-region active-active, known queries, and a team willing to operate it → Cassandra (or ScyllaDB for density). Same workload but no appetite to run a cluster and a single cloud → DynamoDB. Need joins, transactions, or ad-hoc queries → Postgres, sharded when you must. See DynamoDB, PostgreSQL, and the Database Selection case study.

Patterns#

Pattern 1: Time-Bucketed Event Store#

The canonical use. Partition by (entity, time_bucket), cluster by time, TTL the data, TWCS with windows ≈ 1/20th–1/30th of the TTL (e.g., 90-day TTL → 3–7 day windows; keep total windows under ~50). Pick the bucket so the busiest entity's partition stays < 10MB.

bucket = floor(ts / bucket_width)
bucket_width chosen so: max_events_per_entity_per_bucket × row_bytes < 10MB
reads: newest bucket first, walk back until LIMIT satisfied (cap the walk at N buckets)

Pattern 2: Hash-Sharded Hot Buckets#

When the partition key is shared by everything in a time window (a global feed, "all events this hour"), one partition takes the whole write load. Add a synthetic shard: ((hour, shard), ts) with shard = hash(event_id) % 64. Writes spread over 64 partitions; reads fan out 64 single-partition queries in parallel and merge. Choose the shard count from peak write rate ÷ comfortable per-partition rate (~a few thousand writes/s).

Pattern 3: Manual Index Table#

Need to find a notification by ID (Q3, "mark read")? Write notifications_by_id (notif_id → user_id, bucket, created_at) alongside the main row, then update the main row by its full key. Two writes per insert; reads stay single-partition.

Pattern 4: Multi-DC Active-Active#

Each region reads and writes locally; replication is asynchronous; conflicts resolve by last-write-wins per cell. Sticky routing by user keeps a single user's writes in one region most of the time, which makes LWW conflicts rare in practice. On region loss, the global LB shifts traffic; the surviving DC already has (almost) all data. After recovery, run repair on the returning DC before trusting it for reads at lower levels.

Pattern 5: Uniqueness via LWT Side Table#

Keep the main table LWT-free. Claim the unique value in a tiny table with IF NOT EXISTS, then write the main row unconditionally. If the main write fails, a sweeper (or a TTL on unconfirmed claims) releases orphans.

Pattern 6: CDC Out of Cassandra#

Cassandra's CDC writes commit-log segments to a CDC directory per node; consuming it means de-duplicating across RF replicas. Many teams instead emit events from the application (outbox-style) into Kafka, and feed search, analytics, and derived tables from there. Use CDC when the application can't be changed.

Scaling#

The Numbers#

Sizing Example#

Ingest:      40K notifications/s peak, 400 bytes each, RF=3, 2 DCs
Writes/DC:   40K × 3 = 120K replica writes/s  -> at ~15K/node = 8 nodes for writes
Storage:     avg 15K/s (peak 40K) × 86,400 × 90 days × 400B ≈ 47TB raw per replica set
Per DC:      47TB × 3 (RF) ≈ 140TB; with 40% headroom ÷ 0.6 ≈ 233TB
Nodes/DC:    233TB ÷ 2TB/node ≈ 117 nodes  -> storage, not throughput, is the driver
Lever:       compression (LZ4 ~2–3× on JSON payloads) -> ~45–60 nodes/DC

Storage, not request rate, sizes most Cassandra clusters. The two cheapest levers are compression and a shorter TTL — the second is a product conversation.

Scaling Moves in Order#

1. Fix the model — wide partitions and tombstones cause more "we need more nodes" than load does.
2. Tune compaction — wrong strategy can triple read latency or disk use.
3. Add nodes — one at a time (or with care, a rack at a time); each bootstrap streams ~1/N of the data; run nodetool cleanup on existing nodes afterward.
4. Split the cluster — separate workloads with different shapes (time-series vs KV) onto different clusters so compaction and GC don't fight.
5. Add a DC — for latency or region resilience; ALTER KEYSPACE RF for the new DC, then nodetool rebuild from an existing DC.

Failure Modes & Recovery#

1. Tombstone Overwhelm#

• Symptom: a specific query starts timing out; logs show Read N live rows and M tombstone cells; eventually TombstoneOverwhelmingException.
• Root cause: row-level deletes or null inserts concentrated where reads start (queue pattern, "delete read notifications").
• Detection: cassandra.table.tombstone_scanned_histogram p99 > 1,000; warn-log rate.
• Fix: short term, lower gc_grace_seconds on that table only if repair is current, then force compaction; long term, redesign to TTL + TWCS or move the workload.
• Prevention: design review rule — no delete-heavy read-front patterns; ORM unset-values check.

2. Hot or Wide Partition#

• Symptom: one node's CPU and latency spike while peers idle; timeouts for a subset of users.
• Root cause: a celebrity, bot, or global key concentrating reads/writes; or an unbounded partition grown to hundreds of MB.
• Detection: nodetool tablehistograms partition size max; per-node ClientRequest.Latency skew; driver-side per-host latency.
• Fix: rate-limit the abusive key at ingest; add a hash shard to the key (new table + backfill).
• Prevention: size bound in every design doc; alert on max partition > 100MB.

3. Compaction Backlog → Disk Full#

• Symptom: PendingCompactions climbs for hours, SSTable count per table grows, read latency rises, disk usage approaches 90%.
• Root cause: sustained write burst beyond compaction throughput, or STCS major compaction needing ~50% free space that isn't there.
• Detection: cassandra.compaction.pending_tasks > 100 for 30 min; disk > 70%.
• Fix: raise compaction_throughput temporarily, add nodes (streaming also needs disk — act early), drop expired data via TTL, delete snapshots (a common hidden disk eater).
• Prevention: alert at 60–70% disk, not 90%; capacity plan for compaction headroom.

4. Data Resurrection After Missed Repair#

• Symptom: deleted rows (account deletions, GDPR erasures) reappear weeks later.
• Root cause: a replica missed the delete, hints expired (> 3h outage), repair didn't complete within gc_grace_seconds, tombstones were purged elsewhere.
• Detection: repair.last_success_age per table > gc_grace_seconds × 0.7; audit jobs comparing erasure log to live data.
• Fix: re-issue deletes from the erasure log; run full repair; never bring a node back that has been down longer than gc_grace_seconds — wipe and replace it instead.
• Prevention: Reaper with per-table schedules; hard runbook rule on long-down nodes.

5. Coordinator Overload and GC Pauses#

• Symptom: p99 spikes cluster-wide in waves; dropped mutations; nodes flapping DOWN/UP in gossip.
• Root cause: large partitions or big batches materialized on heap; un-token-aware clients making every node coordinate; heap too small for the workload.
• Detection: jvm.gc.pause > 500ms; DroppedMessage.MUTATION > 0; ThreadPool.Native-Transport-Requests.pending.
• Fix: enable token-aware routing in drivers; cap page size (fetch_size ~1–5K rows); move to G1/ZGC and tune; find the big partition.
• Prevention: driver configuration as a shared library default; GC pause SLO per node.

Operational Reality Matrix#

When to Use vs. Alternatives#

When NOT to Use Cassandra#

• Your queries aren't known yet. Early-stage products change access patterns monthly; each change is a new table and a backfill.
• You need cross-entity invariants. "Don't oversell," "balance ≥ 0," "exactly one active session" — LWT handles one partition; anything wider is application-level hope.
• Data is small. Under ~1TB and ~10K writes/s, a well-run Postgres primary with replicas is cheaper and simpler.
• Nobody will own operations. Repair, compaction tuning, node replacement and upgrades are real work. If no one is funded for it, buy managed or pick a different engine.
• The workload is delete-heavy or queue-like. Tombstones will win.

Operational Concerns#

What the On-Call Actually Does#

• Replaces failed nodes (replace_address_first_boot) and watches streaming finish; ~1–4 hours for a 1–2TB node on 10Gbps.
• Watches repair progress in Reaper; restarts failed segments.
• Responds to disk alerts: clears snapshots, checks pending compactions, decides whether to add capacity now.
• Finds the wide partition or tombstone table behind a latency page, then hands the fix to the owning team.
• Runs rolling restarts for config, JVM, and OS patches — one node at a time, waiting for UN status and hint replay.

Key Metrics & Alerts#

Upgrades and Config Changes#

Rolling, one node at a time, in one DC first; never mix major versions longer than the upgrade window; don't run repair or schema changes mid-upgrade; run upgradesstables after a major version jump. Schema changes (ALTER TABLE) propagate via gossip — make them from one place, one at a time, and wait for schema agreement; concurrent schema changes from multiple services are a classic self-inflicted outage.

Interview Application — Staff-Level Plays#

Which Case Studies Use Cassandra#

Every System Design Question Has a Cassandra Moment#

• Chat: "Messages go in Cassandra keyed by conversation and time bucket — append-only, read newest-first, and a bucket bounds the partition even for a 50K-member channel."
• Feed: "Timelines are a materialized answer per user. That's exactly what Cassandra is good at — one partition read per feed load."
• IoT / telemetry: "Device-day partitions, TTL at retention, TWCS — old data disappears as whole files, never as tombstones."
• Anything with money: "Not here. Balances need cross-row invariants; that stays in a transactional store and Cassandra gets the event history."

What Interviewers Probe#

Common Interview Mistakes#

L5 vs L6 vs L7 Responses#

The Staff Cassandra Checklist#

1. "Here are the read queries and their rates — each gets a table."
2. "Partition key is the equality predicate plus a bucket; here's the worst-case partition size."
3. "Clustering order matches how we read — newest first, sliced with LIMIT."
4. "RF=3 per DC, LOCAL_QUORUM reads and writes; here's what that means across regions."
5. "Data expires by TTL with TWCS; we don't delete row-by-row."
6. "Repair runs on a schedule inside gc_grace_seconds, and the platform team owns it."

🎯 Staff Insight: Say what you won't put in Cassandra. "Balances, inventory counts, and anything that needs a uniqueness or ordering guarantee across rows stay in a transactional store. Cassandra gets the high-volume history." The boundary is the signal.

Evaluation Rubric#

Strong Hire Signals

Lean No-Hire Signals

Common False Positives

• Knowing SSTable, memtable, and bloom-filter internals ≠ modeling data correctly.
• Saying "Netflix uses it" ≠ justifying it for this workload.
• Choosing QUORUM everywhere ≠ understanding consistency — in multi-DC it's usually the wrong level.

The Principal Lens#

Why L7 Sees This Problem Differently#

At Staff level, Cassandra is an engine you model correctly. At Principal level it is a long-term operational commitment with a skill requirement most orgs underestimate. The design that's perfect on day one is still running in year five, with three times the data, two reorgs later, owned by a team that didn't build it. The failures that hurt at that point are not modeling mistakes — they are skipped repairs, a node density nobody re-evaluated, and a cluster shared by six teams where one team's tombstones degrade everyone. The L7 question is "Should we be in the business of operating a leaderless database — and if so, how many clusters, for whom, with what guardrails, and what's our exit?"

The Org-Level Fault Line#

One shared Cassandra platform vs. per-domain clusters vs. buying it.

The Principal default: per-domain clusters provisioned and operated by one data-platform team, with a data-model review gate for any new table on a shared cluster, automated repair as part of the platform, and a written decision for which domains should use a managed service instead.

Cost Model#

Assumptions: self-run on cloud VMs with local NVMe, ~$700–1,000/node-month for an 8–16 vCPU / 64GB / 2TB node; 2 DCs; cross-region transfer ~$0.02/GB; loaded engineer ~$250K/yr. Managed estimates assume sustained write-heavy usage. Directional only.

At startup scale, people dominate — half an engineer costs more than the cluster, which is the strongest argument for a managed or different engine. At growth scale, storage density and compression dominate the node count. At enterprise scale, cross-region replication and TTL policy become product-level cost decisions.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

🧭 Principal Move: "Teams can tune compaction, TTLs and node sizes without asking anyone. Partition keys on shared clusters and anything that replicates personal data across regions go through review — those are the decisions we pay for for years."

The Standard I'd Write#

RFC-DATA-014: Using Cassandra on the Data Platform

Scope: Any table on a platform-operated Cassandra cluster.

MUST
  1. Submit the query list, partition key, and worst-case partition size
     (< 100MB, target < 10MB) for review before table creation.
  2. Use LOCAL_QUORUM (or justify otherwise) and token-aware, DC-aware drivers
     from the platform client.
  3. Express retention as TTL; tables with > 1% of writes being deletes need an
     approved exception.
  4. Never store data requiring cross-partition invariants (balances, inventory,
     unique constraints beyond a single LWT partition).
  5. Platform guarantees repair of every table within 70% of gc_grace_seconds and
     alerts on breach.
SHOULD
  6. Use TWCS (or UCS) for TTL'd time-series; LCS only for read-heavy update data.
  7. Tier data older than the hot window to object storage.
  8. Emit domain events via outbox to Kafka rather than consuming Cassandra CDC.

Exceptions: Approved by data-platform lead and the requesting team's director;
reviewed every two quarters.

Success metrics: zero resurrection incidents; zero tables with partitions > 100MB;
p99 read < 20ms per cluster; storage cost per retained TB trending down.

What I'd Tell the VP#

"Cassandra is the right engine for our message and activity history: it handles our write volume across two regions and keeps working if one goes down. The risk isn't the technology, it's that it needs steady care — maintenance jobs that, if skipped, can bring deleted customer data back. Today that work depends on two people. I'm proposing we make it a platform responsibility with automation, a review step for new tables, and a rule that financial data never goes into it. That's roughly one additional engineer and protects both our uptime and our privacy commitments. We'll also revisit, next year, whether a managed service is cheaper than running it ourselves."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "Run repair weekly" — without saying who owns it, how breaches are detected, or what happens when that person leaves.
• "Use a separate table for that query" — correct, but silent on who reviews the next ten tables other teams add to the same cluster.
• "Cassandra is cheaper than DynamoDB at this scale" — without the people cost in the comparison.

In the Wild#

Apple — Scale as an Operational Discipline#

Apple has spoken publicly at Cassandra conferences about running one of the largest deployments in existence — well over 100,000 nodes storing many petabytes — and Apple engineers are among the significant contributors to the Apache project. At that size, the dominant engineering problems are automation: node replacement, repair, upgrades, and fleet-wide configuration.

Staff insight: The largest users invest most in operating Cassandra, not in modeling. In an interview, mentioning who runs repair and replacement is what distinguishes "I've read about it" from "I've carried the pager."

Netflix — Multi-Region, Active-Active#

Netflix adopted Cassandra during its move to AWS for data that needed to stay writable across availability zones and regions, published early benchmarks showing over a million writes per second on a few hundred nodes, and built open-source tooling (such as Priam for backups and token management) around it. Later, Netflix described building a data-access abstraction layer in front of its key-value stores so application teams program against a stable API rather than raw CQL.

Staff insight: The abstraction layer is the Principal lesson: once dozens of teams use a store, the org wants a contract it controls between applications and the engine, so data models and engines can evolve without touching every service.

Discord — Partition Design and Its Limits#

Discord's 2017 engineering post described storing messages in Cassandra with a partition key of channel plus a fixed time bucket, so even very active channels had bounded partitions. Their 2023 follow-up described the problems that emerged at trillions of messages — hot partitions in huge servers, GC pauses, and heavy maintenance — and a migration to ScyllaDB fronted by data services that coalesce concurrent reads of the same hot data, with a substantial reduction in node count.

Staff insight: Both halves are interview gold. The first shows the bucketed-partition pattern done right; the second shows that even a correct model hits hot-partition and operational limits at extreme scale — and that request coalescing in front of the database is often the fix, not just a new engine.

Practice Drill#

Prompt: "Design storage for a connected-car platform: 2 million vehicles each send a 300-byte telemetry record every 5 seconds while driving (assume 20% are driving at peak). Fleet managers view the last 24 hours of a vehicle's trip on a map; data science wants 13 months of history; a vehicle's owner can request deletion of all their data."

Quick Reference Card#

Model:            queries first; one table per access pattern
Primary key:      ((partition cols), clustering cols) -> partition = query's equality predicate
Partition size:   target < 10MB, ceiling 100MB, < 100K rows; bucket by time or hash
Consistency:      R + W > RF;  default RF=3/DC + LOCAL_QUORUM read & write
Cross-DC:         async, ~100ms–1s; LWW per cell by timestamp
LWT:              IF NOT EXISTS / IF col = x; Paxos; ~4–10x write latency; narrow tables only
Writes:           upserts; commit log + memtable; no read-before-write
Deletes:          tombstones; live >= gc_grace_seconds (default 10 days)
Tombstone limits: warn 1,000 / fail 100,000 per read
Repair:           every range within gc_grace_seconds (Reaper); hints window 3h
Compaction:       TWCS for TTL time-series; LCS read-heavy; STCS write-heavy; UCS on 5.x
Node density:     1–2TB classic, ~4TB with care; 30–50% disk headroom
vnodes:           num_tokens = 16 (4.0+)
Indexes:          manual index tables > SAI (partition-restricted) > 2i; avoid MVs
Throughput:       ~10–25K replica writes/s per node; storage usually sizes the cluster

RED FLAGS
  - Partition key with no bound (user_id alone for an event stream)
  - Queue / delete-heavy workload
  - ALLOW FILTERING or 2i in a request path
  - QUORUM (not LOCAL_QUORUM) in multi-DC
  - Counters for anything that's retried
  - No repair schedule, or a node returning after > gc_grace_seconds
  - Balances or inventory in a LWW store