Technologies referenced in this case study: Cassandra · Redis · Apache Kafka · DynamoDB

Related case studies: Real-Time Updates · Notification System · Message Queues · Blob Storage · File Sync · Real-time Updates pattern

How to Use This Case Study#

Organized for interview use first, reference second. Read once end to end, then return to weak spots.

Naive design (server-side history table, client polls, server-assigned timestamps):
t=0:       Phone reconnects after 9 hours. 1,400 messages across 60 chats waiting.
t=+1s:     Client asks "give me everything since my last timestamp." Clock skew between
           two chat servers means 30 messages have timestamps earlier than the client's
           last-seen value. They are never delivered.
t=+3s:     Server pushes 1,400 messages; the connection drops at message 900. Client
           reconnects and re-requests from its last timestamp: 900 duplicates render.
t=+5s:     Group messages appear out of order — replies before the questions.
t=+10s:    Sender's phone still shows one grey tick for messages delivered hours ago,
           because receipts were sent to a device that has since been replaced.

Staff design (per-device inbox with sequence numbers, client IDs, acks):
t=0:       Same reconnect.
t=+1s:     Client says "my inbox cursor is 88,214." Server streams 88,215 onward in
           pages of 100. Order within each chat follows the chat's sequence number.
t=+3s:     Connection drops at 900. Client acked through 88,900 in batches; resumes
           from 88,901. Client-generated message IDs dedupe anything re-sent.
t=+4s:     Server deletes acked messages from the inbox (it is a relay, not an archive).
t=+5s:     Delivery receipts flow back to senders' devices as their own inbox entries.

Executive Summary

If you only read one section, read this. Everything else in the case study elaborates on the contrast and the positions below.

What This Interview Actually Tests#

Chat is not a WebSocket question. Everyone can hold a socket open.

It is a store-and-forward delivery problem with per-conversation ordering, end-to-end acknowledgments, and — under E2E encryption — a server that must route what it cannot read. It tests:

• Whether you define delivery semantics precisely: at-least-once transport + client dedupe = exactly-once display
• Whether you know who assigns order (a per-chat sequencer) and why wall-clock timestamps fail
• Whether you treat the server as a relay with an inbox, not an archive, and know which intent that implies
• Whether you understand what E2E encryption moves to the client: group fan-out, multi-device sync, search, backups, abuse detection

The key insight: The server's job in WhatsApp-style chat is to hold an encrypted envelope until every intended device has acknowledged it, then forget it. Every hard design question — ordering, receipts, groups, multi-device — is a question about who holds which cursor and who can read what.

The L5 vs L6 vs L7 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll design private E2E messaging — WhatsApp's intent. The server is a relay that holds encrypted envelopes per device until they're acked. That choice pushes group fan-out encryption, multi-device sync and search to the client, and I'll call out each place it does. If you want Slack-style history and search, that's a different system and I'd design the storage completely differently."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: these are well-known, publicly documented systems; citing them shows you understand how production chat actually evolved.

WhatsApp — Erlang Relay, Signal Protocol, Multi-Device#

WhatsApp's backend is famously built on Erlang/BEAM (on FreeBSD in its early years), and its engineers publicly described handling ~2 million concurrent TCP connections on a single server in 2012. WhatsApp completed rollout of end-to-end encryption using the Signal Protocol in 2016, and in 2021 published its multi-device architecture, in which each linked device has its own identity key and senders encrypt per device, so companions no longer depend on the phone being online. WhatsApp states that undelivered messages are deleted from its servers after 30 days. It was also widely reported to have served several hundred million users with a team of ~50 engineers at the time of its 2014 acquisition.

Staff insight: WhatsApp's design is a lesson in doing less on the server: a relay that holds encrypted envelopes, a runtime built for millions of lightweight connection processes, and complexity pushed into clients where E2E requires it. Say "the server is a relay" and you've made the most important decision.

Discord — Server-Side History at Trillion-Message Scale#

Discord has written publicly about storing messages in Cassandra and later migrating to ScyllaDB, with messages partitioned by (channel_id, time bucket) and ordered by Snowflake IDs (time-sortable 64-bit IDs). Their posts describe hot partitions from very large, busy channels and the operational pain of compaction and GC pauses that motivated the move.

Staff insight: This is the system-of-record intent. The partition key embeds a time bucket because unbounded channel partitions become hot and huge. Contrast it with WhatsApp: same product category, opposite storage philosophy, because the intent is different.

Facebook Messenger — Iris, a Per-User Ordered Update Queue#

Facebook Engineering described Iris (2014) for Messenger: a totally ordered queue of messaging updates per user, with each device keeping a pointer (sequence ID) into it. Devices sync by asking for everything after their pointer, which made mobile sync fast and reliable over lossy networks.

Staff insight: "Per-user ordered queue + per-device cursor" is the generalizable pattern. It turns sync into one integer comparison and makes resume-after-disconnect exact. Name it in the interview.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: clients own encryption; the core routes opaque envelopes. The Router asks the chat's Sequencer for the next seq, looks up recipient devices (1:1 or group), and appends one entry per device to that device's inbox before acknowledging the sender. If the device is online (per the session registry), the gateway pushes immediately; otherwise the push sender wakes the app. The inbox is the only durable message state, and entries are deleted when the device acks.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: candidates spend 15 minutes on WebSocket handshakes and load balancer stickiness, then hand-wave "and messages are stored in Cassandra." The transport is two sentences. The phases below get you to the inbox, ordering and E2E by minute 12.

Phase 1: Requirements & Framing (2–3 min)#

Functional core in one breath:

"1:1 and group messages with text and media, delivered in real time when online and stored until delivered when offline, with sent, delivered and read receipts, on multiple devices per user."

Then the framing question that decides the design:

"The big fork: is the server the system of record — history, search, compliance — or a relay that holds messages until delivered? For WhatsApp it's a relay with end-to-end encryption. I'll design that, and I'll call out what E2E pushes to the client."

Then numbers:

"~2B users, ~500M–1B concurrently connected at peak, ~100B messages/day — ~1.2M/s average, ~3–5M/s at peak. Average group message fans out to ~5–10 recipient devices, so inbox writes are ~5–10M/s at peak. Online delivery p99 under ~500 ms in-region. Offline retention 30 days."

🎯 Staff Move: The relay-vs-record question in the first 60 seconds is the level signal. It tells the interviewer you know there are two chat systems and you are choosing one on purpose.

Phase 2: Core Entities & API (1–2 min)#

• Device: device_id, user_id, identity_public_key, prekeys[], push_token
• Chat: chat_id (1:1 = hash of sorted user IDs; group = generated), last_seq, owner_shard
• Envelope: msg_id (client-generated UUIDv7), chat_id, seq, sender_device, recipient_device, ciphertext, type ∈ {message, receipt, key_change, membership}, server_ts
• InboxEntry: (device_id, inbox_seq) → envelope; deleted on ack
• GroupMembership: group_id → member user IDs → their device IDs; version

Socket frames (binary, compact):

→ SEND   {msg_id, chat_id, envelopes: [{recipient_device, ciphertext}], membership_version}
← ACK    {msg_id, seq, server_ts}                 # single tick: durably in all inboxes
← DELIVER{inbox_seq, envelope}                    # pushed or fetched
→ RECV_ACK {up_to_inbox_seq}                      # cursor advance; server deletes ≤ cursor
→ SYNC   {after_inbox_seq, limit: 100}            # reconnect / backfill
→ RECEIPT{msg_id, chat_id, kind: delivered|read}  # travels back as an envelope

🎯 Staff Move: "Notice the sender sends one ciphertext per recipient device — the server can't fan out plaintext because it doesn't have any. And the recipient acks a cursor, not individual messages, so one ack covers a batch of 100."

Phase 3: High-Level Architecture (≤ 5 min)#

Three sentences:

"Gateways hold sockets and nothing else. The Router gets a sequence number from the chat's sequencer, appends one inbox entry per recipient device, acks the sender only after those appends are durable, then pushes to online devices or wakes offline ones through APNs/FCM. Devices ack a cursor; the server deletes everything up to it."

Phase 4: Transition to Depth#

"Transport is standard — persistent socket plus OS push. The parts I'd want to go deep on are: how the inbox and acks give exactly-once display, how ordering works in groups, and what E2E does to groups and multi-device. I'd start with the inbox because everything else builds on it."

Phase 5: Deep Dives (25–30 min)#

Exactly-once display, as you'd say it:

"The sender's client generates msg_id before sending and retries with the same ID until it gets an ACK. The Router dedupes by (sender_device, msg_id) for a short window so retries don't create duplicate inbox entries. If a duplicate still slips through — say, an ACK was lost after the append — the recipient's local database has a unique constraint on msg_id and drops it. Transport is at-least-once; display is exactly-once."

Phase 6: Wrap-Up (2–3 min)#

"To summarize: a relay with durable per-device inboxes, at-least-once transport with client IDs for exactly-once display, a per-chat sequencer for order, receipts as messages, fan-out on write with sender keys for groups up to ~1K, and per-device keys for multi-device. What I'd monitor first: online delivery p99, inbox backlog age, and reconnect rate per gateway. What I'd build next: client-to-client history transfer for new devices and encrypted backups. What I skipped: channels/broadcast to millions, which is a fan-out-on-read product."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Chat is everybody's first "real-time" design, which makes it a strong discriminator: nearly every candidate produces something that works in a demo, and the level shows in how precisely they define "delivered" and where they put state.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"When the sender sees two ticks, exactly which device has exactly which bytes?"

Answering this precisely forces every decision:

• One tick = the envelope is durably appended to every recipient device's inbox (replicated). Not "the server received it."
• Two ticks = at least one (WhatsApp: all, in the multi-device model it's per-device and folded) recipient device acked receipt.
• Blue ticks = the recipient opened the chat and the client sent a read receipt — which the user may disable.

🎯 Staff Move: "I'll define the ticks precisely first, because each one is a durability promise. One tick is my promise that the message survives any single server failure. I don't send it until the inbox append is replicated."

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Intent 1 — Private E2E messaging. The server is an untrusted relay. It sees metadata (who, when, size) but not content. Its durable state is per-device inboxes of undelivered envelopes. Clients own history, search, and — for groups — encryption fan-out. Scale is dominated by connection count and inbox write amplification, not by stored history.

Intent 2 — Community/workspace chat. The server is the system of record: history is the product (scrollback, search, pinned messages, integrations, bots). Channels can have 100K+ members, so fan-out on write is impossible for large channels; readers pull from a per-channel log. Storage is partitioned by (channel, time bucket). Content moderation and search are server-side.

Intent 3 — Regulated messaging. Retention and eDiscovery are hard requirements: messages must be retained immutably for years and exportable. The server must be able to read content (or hold escrowed keys). Deletion is governed by policy, not user action.

2.2 When NOT to Use This Design#

🎯 Staff Move: "If this were an in-app support chat with a few thousand concurrent users, I'd use a Postgres table and long polling and ship it in a week. The relay-with-inboxes architecture earns its complexity at hundreds of millions of connected devices and when the server must not read content."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Fault Lines#

3.1 Fault Line 1: Relay vs System of Record#

The arithmetic that settles it: at 100B messages/day, a system-of-record design stores ~36 PB/year raw before replication. A relay stores only the backlog — if 95% of messages are delivered within minutes and the tail is bounded to 30 days, the inbox store holds on the order of days of the offline fraction, typically 1–2 orders of magnitude less.

The Staff default: relay with delete-on-ack and a 30-day bound for undelivered envelopes; history on devices; optional encrypted backup as a separate product surface with its own key-management design.

When to deviate: Intent 2 or 3. If the product needs server search, history on any device, or compliance retention, you are building a different system — say so.

🎯 Staff Move: "The cheapest byte to protect is the one we don't keep. With E2E we can't read history anyway, so keeping it server-side buys us cost and risk but no features."

3.2 Fault Line 2: Ordering — Sequencer vs Timestamps vs Causal#

How the sequencer works:

# chat owner shard = hash(chat_id) % N  (consistent hashing; see foundations)
on SEND(chat_id, msg_id, envelopes):
    if dedupe.seen(sender_device, msg_id): return previous ACK
    seq = chat.last_seq + 1                  # in-memory, owned by this shard
    persist: chat.last_seq = seq  AND  inbox appends for each envelope   # one durable batch
    ack sender {msg_id, seq}

Failover rule: the new owner reads last_seq from durable storage and fences the old owner (epoch number in every write), so two owners can never issue the same seq. A client seeing a gap (seq 41 then 43) waits briefly (~2 s) for 42 or requests a re-sync; gaps can also be legitimate (a message addressed only to other devices).

When to deviate: very large groups where one sequencer is too hot — split into the community/channel model with a partitioned log per channel. For 1:1 chats, a lighter option is ordering per (sender → recipient) direction, since cross-direction interleaving within milliseconds rarely matters to humans.

🎯 Staff Move: "Order is a promise per chat, never across chats. One sequencer per chat, fenced by epoch on failover. Timestamps are for the UI label, not for sorting."

3.3 Fault Line 3: Delivery Semantics and Receipts#

Receipts are just messages traveling the other direction. That gives them durability, ordering and multi-device delivery for free — and avoids a separate "receipt status" database that would need to be updated ~3× per message (sent, delivered, read) × devices.

Receipt cost control:

• Batch: one read receipt covers "read up to seq N" in a chat — not one per message.
• Groups: delivered/read receipts in groups are aggregated client-side into "read by 14 of 20"; for large groups, receipts can be sampled or computed on demand.
• Privacy: read receipts are user-configurable; turning them off means the client simply doesn't send them.

When to deviate: community chat doesn't need per-recipient delivered receipts at all — "unread count" from a per-user read cursor is sufficient.

🎯 Staff Move: "I don't promise exactly-once delivery — nobody can over a mobile network. I promise at-least-once delivery and exactly-once display, and I can explain every step that makes that true."

3.4 Fault Line 4: Group Fan-Out — Write vs Read, and the E2E Cost#

For a group of M members with an average of d devices each:

The server-side optimization with sender keys: store the ciphertext once in a message blob keyed by (chat_id, seq), and write only small pointers (~50 B) into each member device's inbox. That cuts inbox bytes by ~20× for a 1 KB message.

Membership consistency matters for security: a SEND carries the sender's membership_version. If the server's version is newer (someone was added or removed), the Router rejects with stale_membership, and the client fetches the new member list, distributes/rotates sender keys, and resends. That prevents a removed member from receiving new messages and ensures a new member gets the key.

The cap is a design constraint, not a product whim. At 1,024 members × ~2 devices, one message = ~2K inbox pointer writes. A busy group sending 1 msg/s is 2K writes/s — fine. A 100K-member group would be 200K writes per message; that is a broadcast product.

🎯 Staff Move: "The group size cap is where the write amplification and the rekey cost meet. Above ~1K, I'd stop pretending it's a group and build a channel: fan-out on read, admin-only posting, no per-member receipts."

3.5 Fault Line 5: Multi-Device — Primary Relay vs Per-Device Endpoints#

Mechanics of per-device:

• Each user has a device list signed by the primary device; senders verify it so the server can't silently add an eavesdropping "device."
• A 1:1 message is encrypted for each of the recipient's devices and each of the sender's own other devices (so your laptop sees what you sent from your phone).
• A new companion gets recent history by encrypted transfer from the primary device, not from the server.
• Linking is a QR scan on the primary — a physical-presence check.

When to deviate: non-E2E products should just use server history — per-device crypto fan-out buys nothing without E2E.

🎯 Staff Move: "With E2E, a 'user' is really a set of devices, and the device list is security-critical. If the server could append a device, it could read messages. So the device list is signed by the user's primary device and verified by senders."

4. Failure Modes & Operational Reality#

4.1 Reconnect Storm After a Gateway Fleet Deploy#

t=0        Rolling deploy restarts 10% of gateways at once (misconfigured batch size).
t=+1s      ~80M sockets drop. Clients reconnect immediately (no jitter in one app version).
t=+5s      TLS handshakes 40× normal; CPU saturated on remaining gateways.
t=+10s     Session registry writes spike 50×; registry p99 2 ms → 900 ms.
t=+30s     Router can't find sessions; treats online devices as offline; OS push spikes
           to 30M/min; APNs/FCM throttle us.
t=+3min    Messages delivered with 1–2 minute delays; "WhatsApp down" trends.

• Detection: gateway.reconnects_per_s, tls.handshakes_per_s, registry.write_latency_p99, push.throttled_total, delivery.latency_p99.
• Blast radius: global if the deploy is global; cellular-region-specific for carrier outages.
• Mitigation: halt the deploy; server-side reconnect hints (retry_after randomized 0–60 s); prioritize handshake capacity; TLS session resumption.
• Prevention: drain gateways gracefully (send reconnect_in with jitter before closing); deploy batch ≤ 1–2% of connections; client jittered backoff enforced in all versions with a minimum-version policy.
• Owner: Edge/connection platform.

4.2 Inbox Store Partition Hot Spot#

• Scenario: a bot account or a very active 1K-member group floods inboxes; or a celebrity with millions of contacts triggers profile/status fan-out.
• What breaks: inbox partitions keyed by device are fine individually; the sequencer for one very busy group is hot (~thousands of messages/s on one chat owner).
• Detection: sequencer.ops_per_s_max{shard}, inbox.append_latency_p99.
• Mitigation: per-chat and per-sender rate limits (see Rate Limiting); spam/abuse throttles on metadata (E2E means content can't be inspected server-side).
• Owner: Messaging core (sequencer), Integrity team (abuse limits).

4.3 Lost Message After Inbox Failover (The Silent One)#

t=0        Inbox replica set primary fails. Async replication lag was 400 ms.
t=+2s      Replica promoted. Envelopes appended in the last 400 ms — already ACKed to
           senders (one tick) — are gone.
t=+∞       Senders see one tick forever. Recipients never see the messages. No error.

• Detection: inbox.acked_but_missing — hard to detect directly; proxy: senders' clients report "one tick older than 24 h while recipient was online" via telemetry (client.stuck_single_tick_total).
• Blast radius: messages in the replication window for affected devices.
• Mitigation: ack the sender only after quorum (synchronous) replication of the inbox append; accept ~2–5 ms extra latency.
• Prevention: make "one tick = quorum-durable" a written invariant; chaos-test primary failure under load monthly.
• Owner: Messaging core / storage team.

4.4 Prekey Exhaustion#

• Scenario: a device offline for weeks receives many new-session requests (new contacts, group adds). Its uploaded one-time prekeys are consumed.
• What breaks: senders fall back to the signed prekey (weaker forward-secrecy properties for the first message) or fail session setup.
• Detection: keys.prekeys_remaining{device} low; keys.fallback_to_signed_prekey_total.
• Mitigation: clients top up to ~100 prekeys when below a threshold on each connect; server alerts on devices near zero.
• Owner: Key directory team.

4.5 OS Push Degradation#

• Scenario: APNs or FCM latency rises to minutes or throttles us.
• Blast radius: backgrounded/offline devices — messages wait in inboxes (not lost), delivered on next app open.
• Detection: push.send_latency_p99{provider}, push.throttled_total, delivery.latency_p99{device_state=background}.
• Mitigation: collapse pushes per device (one wake-up for many messages), priority for 1:1 over group, back off per provider guidance.
• Owner: Notifications platform (see Notification System).

4.6 New Year's Eve Spike#

• Scenario: 3–5× message volume within ~5 minutes of midnight in each timezone, dominated by group and media messages.
• Mitigation: capacity for 5× per-region peak on sequencers, inboxes and media upload; media uploads prioritized below text; follow-the-midnight capacity shifting across regions.
• Owner: Capacity planning with messaging core.

4.7 Operational Reality Matrix#

🎯 Staff Move: "Look at what's absent: no failure in this table loses a message except the inbox failover — and that's exactly why the sender's single tick waits for a quorum write. Everything else delays delivery; nothing else loses it."

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Cryptography depth ≠ system design. Explaining the double ratchet in detail is impressive and rarely what's being evaluated; the system consequence (per-device fan-out) is.
• WebSocket scaling tricks ≠ delivery design. Knowing epoll limits doesn't answer "where is the message when the phone is off?"
• Kafka everywhere ≠ inbox design. A topic per user doesn't scale to billions of devices; a partitioned inbox store does.
• "Eventually consistent" ≠ acceptable for one tick. The sender's tick is a durability promise.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

6.4 Follow-Up Questions to Expect#

1. "The phone is offline for 40 days. What happens to messages sent on day 1?"
2. "The sender's app crashes after sending but before the ACK. What does the user see on restart?"
3. "Two members of a group send at the same moment. What order does everyone see?"
4. "Someone is removed from a group. How do you guarantee they can't read later messages?"
5. "A user links a new laptop. What does it show?"
6. "A gateway with 1M connections dies. What happens?"
7. "How many writes per second hit the inbox store at peak?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design WhatsApp."

Drill 2: Core Mechanic — The Lost ACK#

Prompt: "Alice sends a message. The server stores it, but the ACK never reaches Alice. Walk me through it."

Drill 3: Make It Concrete — Inbox Sizing#

Prompt: "How big is the inbox store?"

Drill 4: Dependency Down — Push Provider Outage#

Prompt: "FCM is degraded for an hour. What happens?"

Drill 5: Hot Key — The Viral Group#

Prompt: "A 1,024-member group is sending 50 messages/s during a live sports match."

Drill 6: Multi-Tenant — Business Messaging#

Prompt: "Businesses want to message millions of customers through our platform."

Drill 7: Build vs Buy — The Connection Layer#

Prompt: "Should we build our own gateway or use a managed WebSocket service?"

Drill 8: Policy Change Without Outage — Rolling Out Multi-Device#

Prompt: "Move from phone-as-relay to per-device endpoints for 2B users."

Drill 9: Cost#

Prompt: "What dominates the cost of running this, and how would you cut it 30%?"

Drill 10: Multi-Region#

Prompt: "Users in Brazil message users in India. Design the regions."

8. Deep Dive Scenarios#

Deep Dive 1: Peak Traffic — New Year's Eve Delivery Delays#

Context: 00:01 in a large timezone. Online delivery p99 goes from 300 ms to 45 s. Social media says "WhatsApp is down." The on-call escalates to you.

Questions to Surface First:

• Is the delay at the router/sequencer, inbox append, or push to the recipient gateway?
• Are messages with media delayed more than text?
• Are sender ACKs (one tick) delayed, or only delivery (two ticks)?
• Is the session registry healthy?

Typical L5 Approach: Adds router instances and inbox nodes across the board.

Staff Approach: Splits the latency by stage: send_to_ack_ms (sender side) vs ack_to_deliver_ms. Finds ACKs fine but deliver slow: the session registry is overloaded by reconnects from users toggling airplane mode at midnight, so routers misclassify online devices as offline and fall back to push, which is throttled. Mitigations: registry read cache with 30 s TTL on routers; prioritize text over media on gateway egress; raise push collapse window.

Principal Approach: Treats New Year's as a known annual event with a "follow the midnight" capacity plan: registry and gateway capacity shifted region by region, media upload deprioritization pre-configured, and a load test at 5× last year's peak two weeks ahead. Owns the public status communication plan.

Key Takeaway: "Split latency by stage. A slow two-tick with a fast one-tick is a delivery-path problem, not a durability problem."

What clears the Staff bar:

• Stage-level latency decomposition
• Finds the registry → push fallback amplifier
• Pre-agreed degradation order

Deep Dive 2: The Silent Failure — Messages Stuck on One Tick#

Context: Client telemetry shows client.stuck_single_tick_total rising slowly for 2 weeks in one region: ~0.01% of messages never get a delivered receipt even though recipients are active.

Questions to Surface First:

• Did the recipient receive the message (check recipient telemetry for msg_id), or did only the receipt get lost?
• Is it correlated with multi-device users, a client version, or a storage cluster?
• Any failovers in that region's inbox store?

Typical L5 Approach: Assumes receipts are lossy and suggests resending receipts.

Staff Approach: Correlates by recipient device count: nearly all cases are recipients with a companion device linked in the last 30 days. Root cause: the device list cache in the router had a TTL of 1 h; new companions didn't receive envelopes until cache refresh, and the delivered receipt logic waited for all devices. Fix: device list changes invalidate router caches via an event; senders include the device-list version and the router rejects stale versions (like membership versions).

Principal Approach: Establishes "every cache of security-relevant or routing-relevant lists is versioned and invalidated by event, never TTL-only" as an engineering standard. Adds end-to-end synthetic probes (test accounts with multiple devices in each region sending every minute) so silent delivery failures are detected by the platform, not by telemetry trends.

Key Takeaway: "A relay's silent failure is a message that never arrives and never errors. Only end-to-end probes catch it."

What clears the Staff bar:

• Separates lost message from lost receipt
• Finds correlation with a routing-cache TTL
• Adds end-to-end synthetic probes

Deep Dive 3: Large-Customer Onboarding — A Government Uses Groups for Emergency Alerts#

Context: A government agency wants to send emergency updates to 20M citizens via the platform.

Questions to Surface First:

• Is this conversational (replies expected) or broadcast?
• Latency requirement — minutes or seconds?
• Should E2E apply? Who are the "devices" on the sender side?

Typical L5 Approach: Creates many 1,024-member groups and fans out.

Staff Approach: Recognizes broadcast, not chat: use a channel/business-messaging path — the sender posts once to a channel log, subscribers fetch on read or receive batched wake-ups; rate-controlled delivery at a level the inbox and push tiers can absorb (20M × ~1.5 devices = 30M deliveries; at 500K/s, ~1 minute). No per-recipient receipts; aggregate reach metrics instead.

Principal Approach: Weighs the public-interest and political risk of the platform as an emergency channel: SLOs we can publicly commit to, abuse/impersonation of government accounts (verified sender program), and whether to partner or decline. Brings Policy and Legal in before engineering starts.

Key Takeaway: "Don't bend a group feature into a broadcast system. Recognize the intent and use the right fan-out model."

What clears the Staff bar:

• Identifies intent mismatch
• Computes delivery time from throughput
• Protects consumer SLOs

Deep Dive 4: Post-Mortem — Removed Group Member Read Messages#

Context: A security researcher reports that a user removed from a group continued to receive decryptable messages for several minutes.

Questions to Surface First:

• Did the server keep routing envelopes to the removed member's devices, or did the removed member's client keep the old sender key and receive ciphertexts via another path?
• Did senders rotate their sender keys on membership change?
• Were there senders on old client versions?

Typical L5 Approach: Fixes the server to stop routing to removed members immediately.

Staff Approach: Finds two gaps: (1) routers had cached membership for up to 5 minutes; (2) some old clients didn't rotate sender keys on removal. Fix both: membership versions checked on every SEND (stale → reject); sender-key rotation mandatory on removal, enforced by rejecting sends with pre-removal key IDs after a grace period of seconds; old clients below a minimum version blocked from sending in groups.

Principal Approach: Treats it as a security incident with disclosure obligations: coordinates with the researcher (bug bounty), publishes an advisory as appropriate, and makes group-membership semantics part of a formal security model that every client change is reviewed against.

Key Takeaway: "Under E2E, access control is enforced by keys, not by routing. Both must change together."

What clears the Staff bar:

• Distinguishes routing from cryptographic access
• Uses versioning to eliminate cache windows
• Treats as security incident

Deep Dive 5: Multi-Region Expansion — Data Residency for Metadata#

Context: A country requires that messaging metadata for its citizens stays in-country.

Questions to Surface First:

• What counts as metadata — sender/recipient IDs, timestamps, IPs, group membership?
• Does content (already E2E) fall under the rule?
• How are cross-border chats treated?

Typical L5 Approach: Deploys a region in-country and routes those users there.

Staff Approach: Homes those users' devices, inboxes, session registry entries and group memberships in-country. Cross-border chats: the sequencer for a chat is homed based on policy (e.g., the in-country user's region if either participant is in-country); envelopes to foreign recipients leave the country only as delivery requires, with minimal retained metadata. Logs and analytics are aggregated in-country.

Principal Approach: Decides whether to comply, negotiate, or exit the market with Legal and Policy, based on the company's privacy commitments; builds residency as a reusable capability (region-in-a-box with policy-driven data placement) rather than a one-off.

Key Takeaway: "E2E protects content, not metadata. Residency is a metadata-placement problem."

What clears the Staff bar:

• Separates content from metadata
• Policy-driven placement
• Escalates the market decision

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Frame chat as relay vs system of record and commit to one
• Define one tick, two ticks and blue ticks as precise durability and acknowledgment states
• Explain at-least-once delivery with exactly-once display using client IDs and dedupe
• Design per-chat ordering with a fenced sequencer
• Size inbox storage from residency time and connection cost from heartbeats
• Explain group fan-out with sender keys, membership versions and the size cap
• Explain per-device multi-device under E2E with signed device lists
• At L7: take E2E through a cross-functional decision, price the edge fleet, and write the standard

The Bar for This Question#

Mid-level (L4): WebSocket servers, a messages table, online delivery, some offline storage. Timestamps for ordering.

Senior (L5): Adds a queue per user, Cassandra for messages, presence, push notifications, horizontal gateway scaling. Claims exactly-once, sorts by timestamp, and treats E2E and multi-device as footnotes.

Staff+ (L6): Commits to relay with E2E, builds per-device inboxes with quorum durability and cursor acks, uses a per-chat sequencer, treats receipts as messages, explains sender keys and group caps, and designs per-device multi-device with signed device lists. Every failure has a metric and owner. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Exactly-Once Delivery Is Marketing"#

The Staff position: At-least-once + idempotent display.

Why this matters in interviews: Precision about semantics is the Staff-level signal.

10.2 "The Server Should Know as Little as Possible"#

The Staff position: Relay with delete-on-ack for consumer E2E chat.

Why this matters in interviews: It reframes "scale" as "do less."

10.3 "Groups Above ~1K Are a Different Product"#

The Staff position: Cap groups; build channels for broadcast.

Why this matters in interviews: Knowing where a design stops working is Staff judgment.

10.4 "Connections Cost More Than Messages"#

The Staff position: Invest in the edge: graceful drains, jittered reconnects, adaptive heartbeats.

Why this matters in interviews: Most candidates optimize message storage, the smaller cost.

10.5 "E2E Is a Product and Policy Decision First"#

The Staff position: Engineering can implement E2E; the org must choose it.

Why this matters in interviews: Naming the non-engineering owners is a Staff-to-Principal signal.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer designs a messaging system. A Principal engineer sees that messaging is the substrate for half the company's products — consumer chat, business messaging, notifications, support, payments-in-chat, calling signaling — and that the one decision that shapes all of them is who can read the content. Once consumer chat is end-to-end encrypted, every adjacent product must decide whether it lives inside that boundary (and gives up server-side features) or outside it (and must be clearly distinguished to users). The L7 job is to make that boundary explicit, fund the shared edge and inbox platform, and prevent each product from building its own socket fleet and its own half-compatible delivery semantics.

The Org-Level Fault Line#

One messaging platform with a clear E2E boundary vs per-product messaging stacks.

🧭 Principal Move: "Every product on the phone shares one connection and one delivery contract. What differs is whether the server can read the payload — and that is declared per product, reviewed by Security and Privacy, and visible to users. Nobody gets to be ambiguous about it."

Cost Model#

Assumptions: cloud list prices order-of-magnitude; ~0.5–1M connections per gateway host; heartbeats every ~45 s; ~6 device deliveries per message; relay storage (undelivered only); media stored encrypted with TTL; loaded engineer ~$250K/yr.

The line that moves the business case: media. Text inbox storage is a rounding error beside media storage and egress; forwarding dedupe and blob TTLs are worth more than any inbox optimization.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Device Messaging Delivery & Confidentiality Standard (v1)

Scope: Every product that sends data to user devices over a persistent connection or OS push.

Mandatory requirements:

• Products MUST use the shared connection gateway; separate persistent connections from the same app are prohibited.
• Every client-originated write MUST carry a client-generated idempotency ID; recipients MUST dedupe on it.
• Delivery states exposed to users MUST map to defined durability events (e.g., "sent" = quorum-durable in all recipient inboxes).
• Each product MUST declare its confidentiality class: E2E (server cannot read) or server-readable; the class MUST be visible in the UI where users could confuse them.
• Routing-relevant lists (group membership, device lists) MUST be versioned; caches MUST be invalidated by event, not TTL alone.
• Undelivered payloads MUST NOT be retained beyond 30 days; delivered payloads MUST be deleted on ack for E2E products.
• Gateway deploys SHOULD drain ≤ 2% of connections per batch with jittered reconnect hints.

Exceptions: Filed with Messaging Platform, Security and Privacy; reviewed quarterly.

Success metrics: One persistent connection per device across products; client.stuck_single_tick_total ≤ 1 in 10⁵ messages; synthetic end-to-end probe success ≥ 99.99% per region; zero unlabeled server-readable surfaces inside E2E apps.

What I'd Tell the VP#

"Our messaging product's biggest promise is simple: when you see one tick, the message won't be lost, and nobody — including us — can read it. Keeping that promise is cheaper than it sounds because we only store messages until they're delivered. The risks are elsewhere: other teams building their own connections that drain batteries, and business features that blur whether we can read a message. I'm proposing one shared messaging platform with a clear, visible line between private and business messaging. It's mostly reorganizing existing teams, and it protects both our battery reputation and our privacy commitment."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll add E2E with the Signal Protocol." — Correct, but ignores the cross-functional decision and the products it constrains.
• "Per-device inboxes solve multi-device." — Correct, but doesn't own the device-linking security policy or its UX.
• "We'll cap groups at 1K." — Correct, but doesn't decide what product serves the users who need more, or who owns it.

Appendices

Appendix A: Mechanics in Depth#

A.1 Send Path#

on SEND(dev, msg_id, chat_id, envelopes, membership_version, device_list_versions):
    if dedupe.get(dev, msg_id) as prev: return prev
    chat = sequencer.owner(chat_id)                         # fenced by epoch
    if membership_version < chat.membership_version: return STALE_MEMBERSHIP
    for each recipient user: if device_list_versions[u] < directory.version(u): return STALE_DEVICES
    seq  = chat.next_seq()
    blob = store_once(chat_id, seq, ciphertext_or_per_device)   # group: one blob with sender key
    quorum_write([inbox_append(d, pointer(blob, seq, msg_id)) for d in recipient_devices(chat)])
    dedupe.put(dev, msg_id, {seq, ts}, ttl=24h)
    for d in recipient_devices: if registry.online(d): gateway(d).push(d) else push_sender.wake(d)
    return ACK{msg_id, seq, server_ts}

A.2 Receive Path#

on DELIVER(inbox_seq, envelope):
    if local_db.exists(envelope.msg_id): skip          # exactly-once display
    plaintext = decrypt(envelope)                      # session or sender key
    local_db.insert(chat_id, seq, msg_id, plaintext)
    periodically: send RECV_ACK{up_to_inbox_seq}       # batch, e.g., every 100 msgs or 1 s
    send RECEIPT delivered (batched per chat: up to seq)

A.3 Gap Handling#

A client seeing chat seq 41 then 43 waits ~2 s; if 42 doesn't arrive, it asks the server whether seq 42 was addressed to it. If not (e.g., a message to other devices only, or a sender-side retry gap), it records the gap as benign.

Appendix B: Keys and Data Model#

Appendix C: Coordination Mechanisms — Quick Comparison#

Appendix D: Protocol and Client Behavior#

• Heartbeat every ~30–60 s, adaptive per carrier NAT timeout; server closes idle sockets after ~2 missed heartbeats.
• Reconnect: jittered exponential backoff (base 1 s, cap 60 s); server may send reconnect_in hints during drains.
• On reconnect: SYNC{after_inbox_seq} in pages of 100; ack in batches.
• Outbox: unsent messages persisted locally with msg_id; retried until ACK; UI shows clock icon.
• Prekeys: top up to ~100 when below ~20.
• Media: encrypt locally with random key; upload; send message with URL, key and hash.

Appendix E: Observability#

E.1 Core Metrics#

E.2 Critical Alerts#

E.3 Control Plane vs Data Plane#

Data plane: sockets, sends, inbox appends, deliveries, acks. Control plane: gateway deploys and drains, heartbeat policy, group caps, rate limits, retention settings, minimum client versions. Control-plane changes are canaried by region and by client cohort.

Appendix F: Scale Evolution#

F.1 What You Don't Build on Day One#

• Multi-region active routing
• Per-device multi-device (start with one device per account, or server-readable companion sync if not E2E yet)
• Channels/broadcast
• Custom gateway runtime (use a proven WebSocket stack)

What you do build on day one: client message IDs, per-chat sequence numbers, cursor-based sync and delete-on-ack semantics. Those are the protocol commitments that every client in the wild will carry for years.

Appendix G: Fairness, Abuse and Cost#

• Abuse without content: send-rate limits per account and per new account, fan-out limits on forwarding (e.g., forward to at most a handful of chats at once), user reports that include reported messages from the reporter's device.
• Business tenants: per-tenant quotas, templates, quality scores; consumer traffic prioritized on shared tiers.
• Battery fairness: one connection per device across products; collapse pushes.
• Cost attribution: per-product share of connections, deliveries, media bytes; monthly showback.