Technologies referenced in this case study: Redis · Apache Kafka · Cassandra · API Gateways · ZooKeeper & etcd

Start with the pattern, then come here for the system: Real-time Updates — Cross-Cutting Pattern covers SSE vs WebSocket, the connection registry and presence basics. This case study goes deeper on what the pattern page doesn't: the connection gateway fleet, fan-out at scale, reconnect storms, and delivery guarantees.

Related case studies: Chat Messaging · Notification System · Collaborative Editing · Load Balancer · Message Queues · Degraded Mode

How to Use This Case Study#

Without a Staff-level design:
t=0:        Routine deploy restarts 20% of gateway hosts at once.
t=+1s:      2M clients see the socket close. All reconnect immediately.
t=+2s:      Load balancer receives 2M TLS handshakes. Remaining hosts' CPU hits 100%.
t=+5s:      Handshakes time out. Clients retry with a fixed 1s delay.
t=+10s:     Healthy hosts fail health checks under handshake load. More clients disconnect.
t=+30s:     8M clients in a reconnect loop. Auth service sees 400K token validations/s.
t=+4min:    Auth service falls over. Full outage — self-inflicted by a deploy.
t=+45min:   Recovered by blocking traffic at the LB and admitting it region by region.

With a Staff-level design:
t=0:        Deploy drains 2% of hosts per wave, 10 minutes per wave.
t=+0s:      Draining host sends 'reconnect' frame with a random 0–120s delay per client.
t=+2min:    200K clients reconnected over 120s = ~1.7K/s. p99 handshake 40ms.
t=+2min:    Each client resumes from its last sequence number. Zero events lost.
t=+5h:      Deploy complete. Nobody noticed. Nobody was paged.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What This Interview Actually Tests#

Real-time updates is not a WebSocket question. Everyone knows how to open a socket.

It is a stateful-fleet and delivery-contract question that tests:

• Whether you size the system in connections, memory and handshakes — not requests per second
• Whether you know where each event's source of truth lives (hint: not the socket)
• Whether you can design fan-out that survives one channel with 1M subscribers
• Whether you treat reconnection — after deploys, crashes, network blips — as the dominant load event

The key insight: The WebSocket is a cache of a durable log, not the delivery mechanism of record. Push is a latency optimization; the guarantee comes from a per-user or per-channel sequence number and a sync API the client calls on every reconnect. Design it that way and the gateway can be lossy, restartable and boring — which is exactly what you want from a fleet holding 10M connections.

The L5 → L6 → L7 Contrast — Start Here#

The Staff Positions#

The Four Intents#

🎯 Staff Move: "I'll design a general real-time delivery platform for conversational delivery — chat-like semantics where users must not miss events — because it has the strictest contract. Notifications are the same pipeline with dropping allowed. Collaborative sessions need room affinity, which I'll call out as a different routing mode. Live broadcast I'll handle as a hot-channel special case."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Real-time fleets at scale converge on the same shape. Citing them shows you've seen the operational reality, not just the protocol.

Slack — Gateway Servers, Channel Servers, and an Edge Cache#

Slack has publicly described its real-time architecture: stateful Gateway Servers hold client WebSockets and subscribe on behalf of their clients; Channel Servers own channels via consistent hashing and fan messages out to subscribed gateways; separate services handle presence; and an edge cache ("Flannel") serves the boot/state payload clients need on connect, so reconnects don't hammer the primary databases.

Staff insight: The separation of "who holds the socket" (gateway) from "who owns the channel" (channel server) is the Fault Line 1 and 2 answer in production. And Flannel exists because reconnect is the expensive path — the boot payload, not the socket, is what melts backends.

Discord — Per-Guild Processes on the BEAM#

Discord runs its real-time gateway on Elixir/Erlang, with a process per guild (server) that fans out to member sessions. Their engineering blog has described scaling to millions of concurrent users and the special engineering needed for very large guilds — where a single guild's fan-out becomes the bottleneck — as well as work to compress gateway traffic to cut bandwidth.

Staff insight: Discord's big-guild problem is the hot channel fault line. The general design works until one channel has hundreds of thousands of members; then fan-out for that one channel needs its own architecture (relays, lazy member lists, sampled presence).

Netflix — Zuul Push#

Netflix open-sourced Zuul Push, a push notification service that holds persistent WebSocket/SSE connections for millions of devices. Their public talks describe the operational lessons: connections that live too long pin load to old hosts and make deploys dangerous, so they cap connection lifetime with randomization to continuously rebalance, and they emphasize that the herd of reconnecting clients is the main scaling risk.

Staff insight: Randomized connection lifetimes turn a rare, catastrophic mass reconnect into a constant, gentle trickle. That's a Staff move: convert a tail-risk event into routine background load you're always sized for.

(Also worth knowing: WhatsApp publicly reported handling around 2M concurrent connections on a single Erlang server in 2012. It proves density is achievable; it doesn't mean it's the right blast radius for you.)

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: Writes go to the durable path first — the Write API assigns a per-channel sequence number and persists before anything is pushed. The log feeds the subscription router, which knows which gateways have members of each channel and sends each event only there. Gateways are deliberately thin: they hold sockets, check resume tokens, and forward frames. On any reconnect, the client calls the Sync API with its last sequence number and fills gaps from the store. Backgrounded mobile clients are served by OS push from the same log.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 15 minutes on WebSocket vs SSE vs long polling. The interviewer assumes you know. Pick one in 30 seconds and spend the time on the fleet, fan-out, reconnects and the delivery contract.

Phase 1: Requirements & Framing (2–3 minutes)#

Functional scope in one breath:

"Clients keep a live connection and receive events for the channels, conversations and objects they care about within about half a second. Clients can also send small events — typing, acks, presence — upstream."

Then the non-functionals that drive the design:

"The key question is the delivery contract. Can a user miss an event? For chat-like delivery, no — so I'll make the socket a best-effort accelerator over a durable, sequenced log, and guarantee completeness through sync on reconnect. Second, this is a stateful fleet: I'll size it in connections and reconnect handshakes, not QPS."

Commit to numbers:

"Assume 50M DAU, 10M peak concurrent connections, 1M published events/s, average fan-out 10 — so ~10M deliveries/s. Push latency p99 under 500ms. And one hot-channel case: a live event with 2M subscribers."

🎯 Staff Move: "The transport choice is the least interesting part of this problem. I'll use WebSockets for foreground clients and OS push for background, and spend our time on what breaks: fan-out, reconnects and the delivery guarantee."

Phase 2: Core Entities & API (1–2 minutes)#

• Connection (conn_id, user_id, device_id, gateway_host, connected_at, resume_token)
• Subscription (conn_id, channel_id) — derived from membership, not stored per connection durably
• Event (channel_id, seq, type, payload_ref or small payload, created_at)
• Cursor (client-side: last_seq per channel, or a per-user inbox seq)

API surface:

WS  /connect?resume_token=…           → HELLO { conn_id, heartbeat_ms: 25000 }
WS  ← EVENT { channel, seq, type, body }      (server → client)
WS  → ACK { channel, seq } (optional, for metrics only)
WS  ← RECONNECT { after_ms: 0–120000 }         (server-directed drain)
GET /sync?cursors=ch1:1042,ch2:77&limit=500   → { events[], more: bool }
POST /channels/{id}/events { … }               → 201 { seq }   (write path, HTTP not socket)

🎯 Staff Move: "Writes go over HTTP to the write API, not over the socket. The socket is for delivery. That keeps the gateway free of business logic and lets me restart it without losing writes."

Phase 3: High-Level Architecture (≤5 minutes)#

Walk it in 60 seconds:

1. A producer writes an event through the write API, which assigns the next per-channel seq and persists it.
2. The event is appended to the log; the subscription router consumes it.
3. The router looks up which gateways have subscribers for that channel and forwards only to them.
4. Each gateway writes the frame to its local sockets for that channel.
5. If a client was disconnected, on reconnect it calls sync with its cursors and gets anything it missed.

🎯 Staff Move: "That's the whole data path. It's correct on a quiet day. It breaks in three ways at scale: one channel with 2M subscribers, 2M clients reconnecting at once, and gaps nobody notices. Let me take those."

Phase 4: Transition to Depth (1 minute)#

"Three deep dives: fan-out routing including hot channels, the reconnect storm — which is the real peak load of this system — and the delivery contract with sequence numbers. I'd start with reconnects, because that's the failure that turns a routine deploy into a company-wide outage. Preference?"

Phase 5: Deep Dives (25–30 minutes)#

Deep dive 1: Reconnect storms (8–10 min)

"Steady state is easy — idle sockets cost memory, not CPU. The CPU peak is reconnection: a full TLS handshake costs ~1–2ms of CPU, and re-establishing session state costs an auth check plus a boot payload. If 2M clients reconnect in 2 seconds, that's 1M handshakes/s — orders of magnitude over what the fleet is sized for."

Five controls, in order of leverage:

"The last one is what Netflix described for Zuul Push: if connections recycle constantly, a mass reconnect is just a faster version of normal, and new hosts pick up load without a thundering herd."

Deep dive 2: Fan-out (8–10 min)

"Naive design: every gateway subscribes to one Redis pub/sub bus and filters. That's O(events × gateways): 1M events/s × 50 hosts = 50M messages/s, 95% discarded. Instead, the subscription router tracks channel → set of gateways with at least one subscriber, maintained by gateways as clients join/leave. An event goes to ~k gateways where k is the number of hosts with members — for a 10-person chat, usually 1–10."

"Hot channels flip the math: a live event with 2M subscribers spread across all 50 hosts. Per-event cost is 50 router sends and 2M socket writes — fine for one event, fatal for 1K events/s (2B writes/s). So hot channels get: server-side coalescing (send the latest score every 500ms, not every change), sampling for comments ('show 20/s of 10K/s'), and a relay tier so the router sends once per region."

Deep dive 3: Delivery contract (6–8 min)

"Every event gets a monotonic seq per channel at write time. The client tracks last_seq per channel. Three cases: if a pushed event is last_seq + 1, apply it. If it's higher, there's a gap — call sync(since=last_seq). If it's lower or equal, it's a duplicate — drop it. On every reconnect, sync all channels the client cares about (batched, one request). This gives effectively-once, in-order visibility without any per-connection durable queue on the gateway."

Phase 6: Wrap-Up (2–3 minutes)#

"The design principle is that the socket is a cache of a durable log. That makes gateways stateless enough to kill, which makes deploys and failures routine; the durable path gives the guarantee; and the expensive event — reconnection — is designed for explicitly with jitter, drain spreading, admission control and cheap resumption."

Organizational closer:

"The long-term risk is that every product team pushes through this fleet. I'd run it as a platform with per-topic quotas and a topic registry, and cells so a bad config affects 2% of users. The most important thing I'd publish is the delivery contract — so teams don't put payments status on a best-effort channel and call it real-time."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Most system design prompts are stateless request/response problems where scaling means "add more boxes." Real-time delivery is the canonical stateful prompt: each connection is pinned to a host for hours, so every failure, deploy and scaling action has a reconnection cost. Interviewers use it to find candidates who have carried a pager for a stateful fleet — the ones who know that the dangerous moment isn't peak traffic, it's the deploy at 2pm on a Tuesday.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"A user's phone loses signal for 40 seconds in the middle of a busy group chat. Walk me through, message by message, how their client ends up showing exactly the right conversation — no gaps, no duplicates, correct order."

This single question exposes whether the candidate knows that TCP send buffers lie, that the server can't tell a dead connection from a quiet one until heartbeats fail, and that the only robust answer is sequence numbers plus a sync path.

2. Problem Framing & Intent#

2.1 The Four Intents — Explained#

Activity notifications → best-effort, coalesced

• Constraint: very high volume, low value per event (like counts, "someone's typing")
• Strategy: push without durability for ephemeral events; coalesce per client (at most 1 badge update per 2s); drop under backpressure
• Failure mode: stale badge until the next page load
• Who pays for imperfection: nobody meaningfully — product signs off that ephemeral events are lossy

Conversational delivery → gap-free, ordered

• Constraint: user-visible messages must all appear, in order, within ~500ms when online
• Strategy: durable write + per-channel seq + push + sync on reconnect
• Failure mode: gap if the client doesn't detect missing seq; duplicates if it doesn't dedupe
• Who pays for imperfection: users (missed messages), support volume, trust

Collaborative sessions → affinity, convergence

• Constraint: bidirectional, 50–100ms, shared mutable state per room
• Strategy: route all participants of a document/room to the same session server (consistent hash on room_id); the session server is authoritative for ordering; see Collaborative Editing
• Failure mode: session server loss forces room reload from snapshot + op log
• Who pays: users in that room see a 1–3s freeze

Live broadcast → latest-value, sampled

• Constraint: 100K–10M subscribers per channel, one or few publishers
• Strategy: coalescing, sampling, relay tiers, sometimes CDN-delivered SSE or HTTP polling for the long tail
• Failure mode: hot channel saturates router or gateway CPU
• Who pays: every other channel sharing those gateways, unless isolated

🎯 Staff Move: "These four share a gateway fleet but not a delivery contract. I'd tag every topic with its class — ephemeral, durable, session, broadcast — and the platform enforces different rules per class: durable topics must have sequence numbers and a sync endpoint; broadcast topics must declare a coalescing interval."

2.2 When NOT to Use WebSockets#

• Updates every 30s or slower are fine. Poll. 10M clients polling every 60s is ~170K req/s of cacheable GETs — often cheaper and far simpler than a stateful fleet.
• Server→client only, web clients. SSE gives you resume (Last-Event-ID) and HTTP semantics for free.
• The app is backgrounded. Mobile OSes kill background sockets; use APNs/FCM with a hint payload and sync on open.
• Correctness-critical state (payment status, order confirmation). Deliver a hint over the socket if you like, but the client must read the authoritative state from an API. Never make the socket the system of record.
• Very large payloads (files, images). Send a reference; fetch via CDN. See Handling Large Blobs.

2.3 What the Interviewer Leaves Underspecified#

• Delivery semantics. Can users miss events? Must they be ordered? Per channel or globally?
• Multi-device. Does a user have 3 devices each needing every event, with read-state synced?
• Channel size distribution. Median 8 members, but the tail (the company-wide channel, the live event) decides the design.
• Upstream traffic. Typing indicators and presence can exceed message volume 10×.
• Offline behavior. Mobile push? How long do we retain events for sync — 7 days? 30?
• Geography. Single region or global users connecting to the nearest region?

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: Stateful Gateway vs Stateless Gateway#

The tension: The host holding a socket is the cheapest place to keep per-connection state — subscriptions, unacked buffers, session context. It's also the one place guaranteed to disappear.

Staff default: Thin gateways for delivery. The only state on the gateway is the socket and an in-memory index of which local connections subscribe to which channels — rebuildable in seconds from membership when the client reconnects. Collaborative rooms use a separate session tier with explicit affinity, not the delivery gateway.

When to deviate: Small scale (< 100K connections, one team) — a fat gateway with a single deploy unit is fine and faster to build.

🎯 Staff Move: "I want to be able to kill any gateway at any time with no data loss. That's the test for what's allowed to live on it: the socket, yes; anything that isn't reconstructible on reconnect, no."

3.2 Fault Line 2: Broadcast vs Subscription-Routed Fan-Out#

The tension: Broadcasting every event to every gateway is trivially correct and scales as O(events × gateways). Routing only to gateways with subscribers scales with actual interest but needs a routing layer whose state churns with every connect/disconnect.

Back-of-envelope:

Events/s published:         1,000,000
Gateways:                   50 (at 200K conns each for 10M)
Broadcast deliveries/s:     50,000,000 gateway messages — ~95% dropped on arrival
Routed (avg 3 gateways/ev): 3,000,000 gateway messages
Router state:               channels with ≥1 online member × gateways ≈ 20M entries
Router churn (steady):      ~10M conns / 3h avg lifetime ≈ 1K conn events/s × ~20 channels = 20K updates/s
Router churn (storm):       1M reconnects in 120s ≈ 8K conns/s × 20 = 160K updates/s

Staff default: Channel-owner servers (Slack's channel-server shape) for normal channels, sharded by consistent hash on channel_id — see Consistent Hashing. Channels above a threshold (e.g., > 10K online subscribers or > 100 events/s) are promoted to broadcast class: coalesced, sampled, and relayed.

🎯 Staff Move: "Fan-out is designed for the median channel and special-cased for the tail. The median chat has 8 members on 3 gateways. The tail has 2 million on all of them — and for that one I stop promising every event and start promising the latest state."

3.3 Fault Line 3: Push Guarantees vs Pull Sync#

The tension: You can make the socket reliable (server-side per-connection queues, client acks, retransmit) or keep it best-effort and put the guarantee in a pull path.

Staff default: Best-effort push + sequence numbers + sync. Use per-channel seq when channels are small-to-medium and the client tracks < ~1K channels; use a per-user inbox seq for notification-style feeds where one cursor per client is valuable and fan-out is bounded.

The subtle part — the race between sync and live push: subscribe first, then sync. If you sync first, events written between the sync read and the subscription land in neither. Subscribe, buffer live events, sync, then apply buffered events deduped by seq.

🎯 Staff Move: "I don't make the socket reliable. I make it unimportant. If every push can be lost and the user still sees everything after sync, then gateways can crash, deploy and drain freely — and that operational freedom is worth more than any ack protocol."

3.4 Fault Line 4: Reconnect Speed vs Fleet Protection#

The tension: The individual user wants to reconnect in 100ms. The fleet wants reconnects spread over minutes. Both can't win during a mass event.

Staff default: All three together — full-jitter backoff (base 1s, cap 60s) in every client SDK, server-directed RECONNECT{after_ms} during drains (spread 0–120s), and per-host admission control (token bucket, e.g., 2K new connections/s/host, fast 503 + Retry-After). Plus cheap resume: TLS session tickets and a signed resume token (HMAC, 10-minute TTL) so reconnection doesn't hit the auth service.

While a client is waiting to reconnect, it isn't blind: it can poll the sync API every 10–30s. The UX degrades from 200ms to ~15s, not to "broken."

🎯 Staff Move: "I'm deliberately making some users wait up to two minutes to reconnect after a deploy, because the alternative is that everyone waits forty-five minutes after the fleet falls over. And because sync covers the gap, 'waiting' means a slightly delayed message, not a missing one."

3.5 Fault Line 5: Platform Gateway vs Product-Owned Sockets#

The tension: Every product wants real-time. If each builds its own socket server, the client holds 4 connections and the company runs 4 fleets with 4 reconnect-storm postures. If one platform owns it, one bad config push disconnects everything.

Staff default: One multiplexed platform gateway, deployed in cells (e.g., 50 cells × 200K users), with a topic registry: each topic declares owner team, delivery class, max event rate, payload size cap (e.g., 16 KB), and schema. Publish quotas are enforced at the router. Product teams own their topics; the platform owns the fleet and the contract.

🎯 Staff Move: "The platform owns the socket and the guarantee; product teams own the topic and the payload. A team that wants to push 50K events/s to every user files a registry change and gets a quota review — not a surprise at 3am."

4. Failure Modes & Operational Reality#

4.1 Reconnect Storm After a Gateway Crash — Full Timeline#

Scenario: A memory leak in a new gateway build crashes 10 hosts (2M connections) within 30 seconds of each other at peak.

t=0:       10 gateways OOM. 2M sockets close (clients get RST or nothing).
t=+0–5s:   Clients that saw the close start jittered backoff (0–1s first attempt).
t=+1s:     ~1M reconnect attempts in the first second window (jitter spreads the rest).
t=+1s:     LB distributes to 40 healthy hosts → 25K handshakes/s/host vs 2K admission rate.
t=+1s:     Admission control rejects excess fast with 503 + Retry-After: 5–30s (randomized).
t=+2s:     Healthy hosts' CPU at 70% — handshake work bounded by admission rate.
t=+5s:     Clients with dead TCP (no RST) detect it only via missed heartbeat: +25–50s.
t=+30s:    Autoscaler adds hosts. Resume tokens mean auth service sees < 5K validations/s.
t=+90s:    1.6M reconnected. Sync API p99 = 300ms at 20K req/s (edge-cached boot payload).
t=+3min:   All reconnected. Zero events lost (sync filled gaps). Nobody else disconnected.

Detection: gateway.connections_active (sudden drop by host), gateway.new_connections_per_sec, gateway.admission_rejected_total, sync.requests_per_sec, auth.validations_per_sec.

Mitigation: Admission control and backoff do the work automatically; on-call rolls back the build and ensures the autoscaler isn't adding hosts running the bad build.

Prevention: Canary new gateway builds on one cell for 24h (leaks show up over hours); memory ceiling alerts per host; cap per-host connections so any single crash is ≤ 1% of users.

Owner: Gateway platform on-call. Sync API owner must be looped in — sync is the second system that sees the storm.

4.2 Hot Channel Meltdown#

Scenario: A celebrity joins a public live Q&A channel. 1.5M users subscribe in 10 minutes. The chat rate reaches 20K messages/s.

t=0:       Channel has 50K subscribers; owner server handles it fine.
t=+5min:   800K subscribers across all 50 gateways; 8K msgs/s.
           Owner server: 8K × 50 gateway sends = 400K sends/s → CPU 100%.
t=+6min:   Every other channel hashed to that owner (~2% of all channels) sees 5s delays.
t=+7min:   Gateways: 8K msgs/s × 16K local subscribers each = 128M socket writes/s fleet-wide.
t=+8min:   Gateway CPU saturates; heartbeats delayed; clients time out and reconnect → storm.

Detection: router.channel_events_per_sec top-K, router.owner_cpu, gateway.socket_writes_per_sec, push_to_visible_ms_p99 for unrelated channels (collateral damage signal).

Mitigation: Promote the channel to broadcast class at runtime: migrate it off the shared owner to a dedicated relay, coalesce to at most 20 displayed messages/s with sampling, and disable per-message push in favor of batched frames every 500ms.

Prevention: Automatic promotion thresholds (> 10K online subscribers or > 100 events/s); isolate broadcast-class channels on their own owner pool; per-channel rate caps in the topic registry.

Owner: Platform (automatic promotion); product (decides sampling policy — "users see a sample of messages in huge channels" is a product decision).

4.3 Silent Gaps — The Client That Never Syncs#

Scenario: A client release has a bug: on reconnect it re-subscribes but skips the sync call when the app resumes from background. Users report "missing messages" days later.

Detection: Server-side: sync.requests_per_reconnect ratio by client version (should be ~1.0; buggy version shows 0.3). Client-side: client.gap_detected_total (events arriving with seq > last_seq + 1) and client.gap_filled_total — the difference is the silent loss.

Mitigation: Force-upgrade or server-side flag that sends a SYNC_REQUIRED frame after every resume.

Prevention: The client SDK owns the seq/sync logic — product client code can't skip it; contract tests in CI that kill the socket and assert no gaps.

Owner: Client platform team (SDK). This is why the delivery contract must be implemented once, in a shared SDK.

4.4 Half-Open Connections and the Heartbeat Trap#

Scenario: A mobile carrier's NAT drops idle mappings after 30s. Your heartbeat is 45s. Connections look alive on the server but packets go nowhere.

Detection: gateway.heartbeat_timeouts_total by ASN/carrier; push_ack_ratio (if clients ack optionally) dropping for specific networks; server-side send-buffer growth per connection.

Mitigation: Lower heartbeat to 20–25s (costs 10M / 25s = 400K tiny frames/s fleet-wide — cheap); detect dead connections within 2 missed heartbeats.

Prevention: Heartbeat interval configurable per client platform; per-connection send buffer cap (e.g., 256 KB) — exceeding it closes the connection rather than letting memory grow.

Owner: Gateway platform team.

4.5 Connection Registry Overload During a Storm#

Scenario: User-targeted sends look up user → gateways in a Redis registry. During a 2M-connection storm, every connect/disconnect is a write.

Detection: registry.write_latency_p99, registry.ops_per_sec, router.lookup_miss_total.

Mitigation: Batch registry updates per gateway (every 1s, not per connection); tolerate stale entries (a send to a gateway without the user is a cheap no-op); rate-limit registry writes during storms.

Prevention: Prefer channel-owner routing (gateway registers interest in channels, batched) over per-user registry for most traffic; shard the registry; TTL entries with heartbeat refresh (every 60s) so crashes self-clean without delete storms.

Owner: Platform team. Key lesson: the system tracking the storm must survive the storm.

4.6 Slow Consumers and Head-of-Line Blocking#

Scenario: A client on a 2G connection can't drain its socket fast enough; the gateway's write buffer for it grows, and if writes are synchronous per channel, other clients wait.

Detection: gateway.conn_send_buffer_bytes p99, gateway.write_queue_depth, per-host event-loop lag.

Mitigation: Non-blocking writes with per-connection bounded buffers; on overflow drop ephemeral events first, then close the connection (client will sync on reconnect).

Prevention: Class-aware buffers: ephemeral events are droppable, durable events are recoverable by sync — so the gateway can always drop.

Owner: Platform team.

4.7 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Knowing the WebSocket frame format ≠ designing a fleet. Opcode trivia doesn't predict operational judgment.
• "We'll use a managed service" ≠ design. Fine as a build-vs-buy answer, but the candidate still needs to state the delivery contract and reconnect behavior.
• Redis pub/sub fluency ≠ fan-out design. The question is routing cost at 50+ hosts and hot channels, not the API.
• Mentioning Kafka ≠ durability story. The log is only useful if the client has a cursor into it and a sync path.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• The WebSocket handshake details and frame format. One sentence.
• Message storage schema in depth. (channel_id, seq) clustered key in a wide-column store is enough; link to Chat Messaging.
• Presence internals. Covered in Real-time Updates; mention TTL-based presence and move on.
• Authentication flows. Token at connect, resume token after; that's it.

6.4 Follow-Up Questions to Expect#

1. "How do you know a connection is dead?" — (Heartbeats every 25s, dead after 2 misses; send-buffer cap closes stuck connections.)
2. "Where does ordering come from?" — (Per-channel seq assigned by the write path — a single sequencer per channel, e.g., the channel owner or a DB counter; no global order.)
3. "How big can one host get?" — (Hundreds of thousands to ~1M+ is technically feasible; I'd cap at ~200K for blast radius.)
4. "How do you deploy the gateway?" — (Drain cell by cell, 2% of hosts per wave, server-directed reconnect with 0–120s spread.)
5. "What does the load balancer do?" — (L4 pass-through or TLS termination with session tickets; least-connections balancing — round robin skews toward new hosts badly after storms.)
6. "What if the sync API is down?" — (Clients keep receiving live pushes; gaps are detected and retried with backoff; alert on sync.error_rate.)
7. "How do you rebalance after scaling up?" — (Randomized max connection lifetime of 2–4h moves load gradually; never mass-disconnect to rebalance.)

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design a system to push real-time updates to our users."

Drill 2: Capacity Math#

Prompt: "How many gateway hosts do you need for 10M concurrent connections?"

Drill 3: Make the Delivery Guarantee Concrete#

Prompt: "How do you guarantee a user sees every message?"

Drill 4: The Deploy#

Prompt: "You need to deploy a new gateway version to 65 hosts holding 10M connections. Go."

Drill 5: The Hot Channel#

Prompt: "A live sports channel has 3M concurrent subscribers and 5K comments/s."

Drill 6: Reconnect Storm Math#

Prompt: "A network blip disconnects 4M clients at once. What happens?"

Drill 7: Multi-Tenant Platform#

Prompt: "Five product teams want to push events. How do you let them?"

Drill 8: Build vs Buy#

Prompt: "Why not use a managed real-time service?"

Drill 9: Multi-Region#

Prompt: "Users in Europe complain of 400ms push latency. Go multi-region."

Drill 10: Protocol Change Without an Outage#

Prompt: "We want to switch from JSON frames to compressed binary frames."

8. Deep Dive Scenarios#

Deep Dive 1: The Deploy That Took Down Everything#

Context: A config change to the gateway's TLS settings was pushed to all hosts at once. Every host restarted within 90 seconds. 10M clients reconnected; the auth service collapsed; the outage lasted 38 minutes. You're asked to lead the post-incident work.

Questions to Surface First:

• Why could a config change reach 100% of hosts at once?
• Did clients back off with jitter, or reconnect in lockstep?
• Why did reconnection hit the auth service at all?
• Which systems were on the reconnect path, and what were their storm budgets?

Typical L5 Approach: Adds more auth service capacity and makes clients back off longer. Both help; neither addresses why a single push could restart the whole fleet.

Staff Approach: Fixes each layer on the reconnect path: config changes go through the same cell-by-cell progressive rollout as code; resume tokens remove auth from reconnect; handshake admission control per host; SDK full-jitter backoff verified in tests; boot/sync payload cached at the edge.

Principal Approach: Treats it as an org failure-posture problem: config and code must share one progressive delivery system for every stateful fleet, enforced by tooling. Introduces cells with hard isolation so no single change can touch more than one cell per hour, and schedules a quarterly mass-reconnect game day with auth and sync owners in the room.

Key Takeaway: "In a stateful fleet, the blast radius of a change is measured in reconnects. Design the rollout, not just the recovery."

What clears the Staff bar:

• Maps every system on the reconnect path
• Removes auth from reconnect rather than scaling it
• Puts config under the same rollout discipline as code

Deep Dive 2: Messages Silently Missing#

Context: Support tickets say users "sometimes miss messages." Metrics show push success at 99.99%. No errors anywhere.

Questions to Surface First:

• What does "push success" measure — a write into the kernel send buffer?
• Do clients report gaps? Do they sync on every reconnect, including resume from background?
• Is it concentrated by client version, platform or network?

Typical L5 Approach: Adds retries on push and increases heartbeat frequency. Push "success" was never the problem.

Staff Approach: Instruments the contract, not the transport: client.gap_detected_total, client.gap_filled_total, and sync_per_reconnect by client version. Finds an iOS build skipping sync on foreground resume. Fixes in SDK, forces sync server-side for affected versions.

Principal Approach: Establishes an end-to-end delivery SLO measured from the client (e.g., 99.9% of events visible within 2s, 100% after sync) and makes it the platform's headline metric; server-side "push success" is demoted to a debugging metric. Funds a synthetic client fleet that continuously verifies the contract.

Key Takeaway: "A 99.99% push success rate says nothing about whether users saw their messages. Measure at the client."

What clears the Staff bar:

• Distrusts transport-level success metrics
• Locates the guarantee in client logic and owns it

Deep Dive 3: Onboarding a Massive Tenant#

Context: A large enterprise customer is moving 400K employees onto the product. They have a single all-company channel with 400K members, and every morning at 9am local time ~300K people log in within 20 minutes.

Questions to Surface First:

• How many of the 400K are online simultaneously, and how many events/s does the all-company channel get?
• What happens at 9am — login wave or thundering herd?
• Are they concentrated in one region/cell?

Typical L5 Approach: Adds gateway capacity for 400K more connections.

Staff Approach: Recognizes two separate problems. The morning wave: 300K connects in 20 min is only ~250/s — fine — but boot payload for a 400K-member workspace is huge; serve it from an edge cache and lazy-load member lists. The all-company channel: pre-promote to broadcast class, restrict posting to admins or rate-limit it, and lazy-load member presence rather than fanning it out.

Principal Approach: Negotiates product limits with the customer (announcement-only channels above 50K members), places the tenant across multiple cells to avoid concentrating risk, and prices the dedicated capacity into the contract.

Key Takeaway: "Large tenants don't break the fleet with connections. They break it with one giant channel and one giant boot payload."

What clears the Staff bar:

• Separates connection load from channel fan-out load
• Uses product limits as an engineering tool

Deep Dive 4: Post-Mortem — Hot Channel Took Down a Shard#

Context: A viral live event channel hashed to the same channel-owner server as 2% of all channels. For 12 minutes those channels had 10–30s delays. Write the post-mortem actions.

Questions to Surface First:

• Why wasn't the channel promoted automatically?
• Why did one channel's load affect unrelated channels?
• How long did detection take?

Typical L5 Approach: Adds more owner servers so each owns fewer channels.

Staff Approach: Automatic promotion at thresholds (> 10K online or > 100 events/s) with migration to a dedicated broadcast pool; per-channel CPU accounting on owners; alert on push_to_visible_ms_p99 for a sample of unrelated channels (collateral damage detector).

Principal Approach: Defines "noisy neighbor isolation" as a platform requirement across all shared tiers (router, gateway, sync), with per-tenant/per-channel budgets; reviews every shared tier for the same failure class.

Key Takeaway: "Consistent hashing distributes channels, not load. The tail needs its own lane."

What clears the Staff bar:

• Detects collateral damage, not just the hot spot
• Automates the promotion decision

Deep Dive 5: Multi-Region Expansion#

Context: The company is launching in Asia-Pacific. Leadership wants < 200ms push latency there and regional failover.

Questions to Surface First:

• What fraction of channels span regions?
• Where does per-channel ordering (seq assignment) live?
• Data residency requirements for message content?

Typical L5 Approach: Deploys the full stack in APAC and routes by geography.

Staff Approach: Regional gateways and routers; per-channel home region assigns seq; events replicate to other regions once per event; sync from regional replicas with seq-awareness; regional failover by jittered reconnect to the next region.

Principal Approach: Decides the home-region policy and its residency implications with legal; budgets cross-region egress; sequences the rollout as read-only regional delivery first, then regional writes, with a failover game day before GA.

Key Takeaway: "Replicate events between regions, not deliveries."

What clears the Staff bar:

• Keeps ordering with a clear owner
• Treats failover as a reconnect storm to design for

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Explain why the socket is a cache of a durable log and design seq + sync with gap/duplicate/race handling
• Size a gateway fleet by connections, memory and reconnect handshakes, and choose host size for blast radius
• Design subscription-routed fan-out and promote hot channels to a broadcast class
• Design against reconnect storms: jitter, drain spreading, admission control, resume tokens, randomized lifetimes
• Run deploys as drains, cell by cell
• Define platform vs product ownership with a topic registry and quotas
• Extend the design to multi-region with a home-region sequencer

The Bar for This Question#

Mid-level (L4): WebSocket server, a pub/sub bus, clients reconnect. Works for a demo.

Senior (L5): Adds a connection registry, horizontal scaling, heartbeats, maybe SSE vs WebSocket analysis. Correct at moderate scale; delivery assumed reliable; reconnect storms and hot channels not considered.

Staff+ (L6): Makes the durable log the source of truth with seq and sync; thin gateways that can be killed; fan-out routed by subscription with a hot-channel lane; reconnect designed as the peak load; deploys as drains; ownership split between platform and topic owners. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Reliable WebSocket Delivery Is the Wrong Goal"#

The Staff position: Make the socket best-effort and the sync path authoritative.

Why this matters in interviews: It collapses a sprawling ack/retry design into one clean invariant.

10.2 "Your Peak Load Is a Deploy, Not a Traffic Spike"#

The Staff position: Size for reconnect handshakes/s and design deploys as spread drains.

Why this matters in interviews: It shows you've operated a stateful fleet.

10.3 "Most Apps Should Poll"#

The Staff position: Ask what staleness product will accept before building sockets.

Why this matters in interviews: Knowing when not to build is a Staff-level signal.

10.4 "One Million Connections Per Host Is a Vanity Metric"#

The Staff position: Cap at ~100–200K per host and spend the extra machines on blast radius.

Why this matters in interviews: It reframes capacity as risk management.

10.5 "Presence Is More Expensive Than Messaging"#

The Staff position: Presence is lazy, coarse, and scoped to what's on screen.

Why this matters in interviews: It shows you can identify the hidden amplifier.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

At Staff level, the real-time system is a fleet to design. At Principal level, it's shared infrastructure that every product team will depend on — and therefore the org's single largest correlated-failure risk for user-facing freshness. The L7 questions are: what contract does the platform offer, what does it refuse to carry, how is its blast radius partitioned, and how is its cost attributed so that teams don't treat live updates as free.

The Org-Level Fault Line#

One real-time platform vs per-product real-time stacks. One platform means one socket per client (battery, connection cost), one team that has mastered storms, and one contract. It also means one set of cells whose failure is felt by every product at once and a platform backlog that gates every new real-time feature. The L7 position: centralize the socket, the delivery contract, the SDK and the storm engineering; decentralize topics, payloads and product semantics via self-serve registry and quotas; keep collaborative session servers as a separate, affinity-based tier owned closer to the product.

Cost Model#

Assumptions: ~$0.10/host-hour for a 16-vCPU, 32 GB instance class; 200K connections/host; egress at ~$0.02–0.05/GB blended; engineers at $250K fully loaded.

The surprising line item is egress, not compute: frame compression and coalescing are cost projects, not just performance projects.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Real-Time Delivery Platform Standard (v1)

Scope: Any feature that pushes server events to clients.

MUST:

• Use the platform gateway and client SDK; no product-owned sockets.
• Register every topic with owner, delivery class, max publish rate and payload cap (≤ 16 KB).
• Durable topics MUST assign a per-channel seq on write and expose sync; the socket MUST NOT be the system of record.
• Clients MUST implement full-jitter backoff (via SDK) and honor server RECONNECT delays.
• Fleet changes (code or config) MUST roll out cell by cell.

SHOULD:

• Broadcast-class topics declare a coalescing interval and sampling policy.
• Ephemeral events (typing, presence) be droppable under backpressure.

Exceptions: Collaborative session servers (affinity tier) — reviewed by the platform team; time-boxed.

Success metrics: 99.9% of events visible < 2s at the client; zero unfilled gaps; no single change affecting > 1 cell per hour; one socket per client.

What I'd Tell the VP#

Real-time updates are now part of every product we ship, and they all run through connections that have to be re-established whenever anything changes on our side. Today, a single bad deploy can disconnect every user at once, and our biggest outages have been self-inflicted in exactly that way. I'm proposing we consolidate onto one real-time platform, split it into independent cells so any failure touches a few percent of users, and guarantee — measured on users' devices — that nobody misses a message. It's about six engineers for three quarters, and it removes four duplicate systems and our most common cause of company-wide incidents.

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "Jittered backoff and admission control" — correct, but doesn't address the org-wide correlated risk of one fleet for all products.
• "Seq + sync" — correct, but doesn't make it a published contract with a client-measured SLO.
• "Platform owns the gateway" — correct, but no chargeback, quotas or exception process.

🧭 Principal Move: "The technical design is a solved pattern. What I'd spend my time on is the contract: what the platform guarantees, what it refuses to carry, and how its blast radius is partitioned — because every product team's freshness now depends on one fleet."

Appendices

Appendix A: Gateway Mechanics#

A.1 Connection Lifecycle#

A.2 Gateway Event Loop (pseudocode)#

on_connect(req):
    if not admission.try_take(): return reject(503, retry_after=rand(5,30))
    user = verify_resume_token(req.token) or auth_service.verify(req.jwt)
    conn = Conn(user, device, send_buf_cap=256KB, max_life=rand(2h,4h))
    for ch in membership.channels_for(user):     # cached, batched
        local_index[ch].add(conn)
    router.register_interest_batch(local_index.new_channels())   # flushed every 1s
    conn.send(HELLO{heartbeat_ms=25000})

on_router_event(ch, frame):
    for conn in local_index[ch]:
        if not conn.try_write(frame):             # non-blocking
            if frame.class == EPHEMERAL: drop
            else: conn.close(reason=SLOW)         # client will sync on reconnect

on_tick():
    close conns with 2 missed heartbeats
    for conn past max_life: conn.send(RECONNECT{after_ms=rand(0,30000)})

Appendix B: Data Model and Sequencing#

Why per-channel and not global order: global ordering needs a single sequencer — a throughput ceiling and a SPOF. Users only perceive order within a conversation.

Appendix C: Fan-Out Mechanisms — Quick Comparison#

Appendix D: Client Contract#

• Connect with resume token if present; else JWT.
• Full-jitter backoff: delay = random(0, min(60s, 1s × 2^attempt)); honor Retry-After and RECONNECT{after_ms}.
• Subscribe → buffer → sync → drain buffer (dedupe by seq).
• On gap (seq > last + 1): sync that channel; on seq ≤ last: drop.
• While disconnected > 10s: poll sync every 15s.
• On app background: close socket; rely on OS push hints; sync on foreground.

Appendix E: Observability#

E.1 Core Metrics#

gateway.connections_active{host,cell}      gateway.new_connections_per_sec
gateway.admission_rejected_total           gateway.heartbeat_timeouts_total{asn}
gateway.conn_send_buffer_bytes (p99)       gateway.event_loop_lag_ms
router.owner_cpu{owner}                    router.channel_events_per_sec (top-K)
push_to_visible_ms (p50/p99, client-measured)
client.gap_detected_total                  client.gap_filled_total
sync.rps / sync.p99                        sync_per_reconnect{client_version}

E.2 Critical Alerts#

E.3 Debugging the Silent Failure#

Server metrics can't see missed deliveries. Run a synthetic client fleet (~1K clients across regions and networks) that publishes and subscribes to probe channels, measures publish-to-visible latency, and verifies no gaps — including through forced disconnects.

Appendix F: Scale Evolution#

What You Don't Build on Day One#

• Relay tiers (until a channel exceeds ~10K online members)
• Multi-region sequencing
• Per-connection durable queues (ever, usually)
• Custom binary protocol (negotiate later)

Appendix G: Multi-Tenancy and Cost#

• Chargeback units: connection-minutes (gateway memory) and deliveries (CPU + egress) per topic owner.
• Quotas: per-topic publish rate; per-tenant maximum channel size for open posting; payload cap 16 KB.
• Isolation: broadcast-class channels on their own owner pool; large tenants spread across cells.
• Tradeoff summary: thinner gateways cost a hop (~1–5ms) and buy freedom to kill hosts; smaller hosts cost machines and buy smaller storms; best-effort push costs a sync API and buys stateless gateways. Every one of those trades is worth it above ~1M concurrent.