Technologies referenced in this case study: Apache Kafka · Redis · PostgreSQL · API Gateways

Related case studies and patterns: Distributed Job Scheduler · Message Queues · Rate Limiting · Leaderboard · Auto-Scaling & Capacity · Long-Running Processes · Degraded Mode · Real-time Updates

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once, then return to the sections that match your weak spots.

Without a Staff-level design:
t=0:        Contest opens. 30,000 users read problem 1.
t=+4min:    First wave of submissions — 900 submits/s vs a 150/s steady state.
t=+5min:    Shared FIFO queue depth hits 60,000. Practice users are in the same queue.
t=+9min:    p50 verdict latency = 7 minutes. Users resubmit, doubling queue depth.
t=+12min:   Autoscaler adds hosts; cold container images take 90s each to pull.
t=+20min:   Noisy neighbors inflate wall-clock time; 8% of correct solutions get TLE.
t=+95min:   Contest ends. Leaderboard is wrong. Rejudge of 40K submissions takes 3 hours.
t=+1 day:   Contest declared unrated. Forum thread titled "judge is broken again".

With a Staff-level design:
t=-30min:   Contest pool pre-warmed to 3× forecast peak from registration count.
t=0:        Same 30,000 users, same spike.
t=+4min:    Contest lane absorbs 900/s. Practice lane throttled to 50% capacity.
t=+5min:    p95 verdict latency 6s. Penalty time stamped at ingress, not at verdict.
t=+20min:   Timing measured as cgroup CPU time on pinned cores. False-TLE rate < 0.1%.
t=+95min:   Leaderboard final within 2 minutes. Zero rejudges needed.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What This Interview Actually Tests#

Designing LeetCode is not a queue-and-worker question. Everyone draws a queue and a worker.

It is a trust-boundary and burst-capacity question that tests:

• Whether you treat submitted code as an attacker, not a user
• Whether you plan capacity for a spike you can see coming (contests are scheduled)
• Whether you separate fairness of the verdict from latency of the verdict
• Whether you own what happens when the judge itself is wrong — bad test data, flaky timing, mass rejudge

The key insight: The judge's product is a trustworthy verdict, not fast execution. A 6-second Accepted that users trust beats a 1-second verdict that is wrong 2% of the time. Every design decision — isolation, timing, queueing — should be evaluated by what it does to verdict trust.

The L5 → L6 → L7 Contrast — Start Here#

The Staff Positions#

The Four Intents#

Four intents drive incompatible designs. Name them and commit.

🎯 Staff Move: "I'll design for contest judging, because it forces every hard decision — a known spike, ranking fairness, and deterministic timing. Practice is a strict subset: the same pipeline with a lower priority lane. I'll call out where the Run button and B2B assessments need different policy, not different architecture."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Code execution isolation is a solved-in-public problem. Citing how the hyperscalers isolate untrusted code signals you know the design space is not "Docker vs not Docker."

AWS Lambda — Firecracker MicroVMs#

AWS built Firecracker, an open-source KVM-based virtual machine monitor, to run Lambda and Fargate workloads from mutually untrusting customers on shared hardware. The published design goals are boot times on the order of ~125ms and memory overhead under ~5 MiB per microVM, which makes "one VM per untrusted workload" economically viable where traditional VMs (seconds to boot, hundreds of MB overhead) were not.

Staff insight: Firecracker exists because AWS concluded containers alone were not an adequate boundary between tenants. If AWS will not trust namespaces + seccomp between paying customers, you should not trust it between anonymous internet users submitting C++.

Google gVisor — A User-Space Kernel#

Google open-sourced gVisor, which implements a large part of the Linux syscall interface in a user-space process ("the Sentry") so that sandboxed code never talks directly to the host kernel. It is used for Google's own multi-tenant serverless products. The tradeoff is overhead on syscall-heavy workloads and incomplete syscall coverage.

Staff insight: gVisor is the middle option on the isolation-vs-latency curve — OCI-compatible (so your container tooling works) with a much smaller host attack surface. For a judge where most programs are CPU-bound and do little I/O, the syscall overhead is mostly irrelevant, which makes gVisor unusually well-suited.

IOI / Competitive Programming Judges — isolate and CMS#

The International Olympiad in Informatics' Contest Management System (CMS) uses isolate, a small sandbox built on Linux namespaces and cgroups, designed specifically for measuring CPU time and memory of contestant programs precisely. Large public judges (Codeforces, AtCoder) have long been known among competitors for visible "In queue" backlogs during the busiest rounds — occasionally leading to extended or unrated contests.

Staff insight: Competitive programming judges optimized for measurement precision, not just containment. That's the fault line most candidates miss: a sandbox that is secure but gives a 20% timing variance is still a broken judge.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: The request path is deliberately boring — gateway, submission service, queue, dispatcher, judge host. The Staff content is in the annotations: ingress timestamps (fairness decoupled from lag), weighted lanes (who waits during a spike), microVMs on pinned cores (trust boundary + timing determinism), test data cached locally by content hash (no hot object-store reads during a contest), and a control plane that turns the contest calendar into pre-warmed capacity.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 20 minutes on the CRUD — problems table, submissions table, a REST API — and 3 minutes on the sandbox. The interviewer picked this problem for the sandbox and the spike. Compress the basics to ~10 minutes.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional scope in one breath:

"Users browse problems, write code in one of ~20 languages, click Run against custom input or Submit against hidden tests, and get a verdict. We also host weekly contests with a live leaderboard."

Then spend your time on the non-functionals — this is where the design lives:

"Three things make this hard. First, every submission is untrusted code — I'll treat it as an exploit attempt. Second, contests produce a scheduled 5–10× spike in the first and last ten minutes. Third, the verdict has to be trustworthy: the same code should get the same verdict every time, or rankings are meaningless."

Commit to numbers:

"I'll assume 10M submissions/day on practice — about 115/s average, 300/s peak — plus a weekly contest with 30K participants that peaks around 300 submits/s on top. Each submission is ~100 test executions and ~2 CPU-seconds. Contest verdict p95 under 10 seconds. No outbound network from user code, ever."

🎯 Staff Move: Say the threat model out loud in the first three minutes. "I'm going to assume the submitter is an attacker" reframes the entire interview from a queue problem to a security-and-capacity problem — which is what the interviewer wanted to test.

Phase 2: Core Entities & API (1–2 minutes)#

Name the nouns, don't draw an ER diagram:

• Problem (id, version, time_limit_ms, memory_limit_mb, checker_type)
• TestSet (problem_id, version, content_hash, cases[]) — immutable, versioned
• Submission (id, user_id, problem_id, contest_id?, language, source_hash, ingress_ts, lane)
• Verdict (submission_id, testset_version, status, cpu_ms, mem_kb, failed_case_idx) — keyed by both

API surface:

POST /submissions            { problem_id, language, source, contest_id? } → 202 { submission_id }
GET  /submissions/{id}       → { status: QUEUED|RUNNING|DONE, verdict?, cpu_ms?, mem_kb? }
POST /runs                   { problem_id, language, source, stdin } → 202 { run_id }
GET  /contests/{id}/standings?page=1

Results are asynchronous. The client polls GET /submissions/{id} with backoff (500ms → 1s → 2s) or subscribes via SSE for the contest page.

🎯 Staff Move: "Verdict is keyed by (submission_id, testset_version), not submission_id alone. That one decision makes rejudging idempotent and auditable — we never overwrite a verdict, we add a new one against a new testset version."

Phase 3: High-Level Architecture (≤5 minutes)#

Walk it in 60 seconds:

1. Gateway authenticates and applies per-user quotas (Submit 6/min, Run 10/min).
2. Submission service stamps ingress_ts, stores source in object storage, writes the submission row, enqueues to the right lane.
3. Dispatcher pulls with weights (contest 70 / practice 25 / run 5 during a contest; 0 / 80 / 20 otherwise).
4. A judge host takes a warm microVM, injects source, compiles, streams test cases via stdin, compares outputs outside the sandbox, destroys the VM.
5. Verdict goes to Postgres (durable) and Redis (status cache the client polls).

🎯 Staff Move: "This is the design that works on a Tuesday. It breaks in three places on contest day — the trust boundary, the spike, and timing variance. Let me take those in order of risk."

Phase 4: Transition to Depth (1 minute)#

"Three deep dives matter here: how the sandbox actually contains hostile code, how we absorb a scheduled 10× spike without making rankings unfair, and how we keep verdicts deterministic on shared hardware. I'd start with isolation because a failure there is a security incident, not a slow page. Which would you like?"

If the interviewer has no preference, lead with isolation — it is the most distinctive part of this problem and the one where L5 answers are thinnest.

Phase 5: Deep Dives (25–30 minutes)#

For each: state the tradeoff → pick a position → quantify the cost → name who absorbs it.

Deep dive 1: The sandbox (8–10 min)

"I layer five controls, each stopping a different attack class:"

"The first four are containment. The fifth is the blast-radius limiter. If a submission escapes the container via a kernel bug, it lands in a single-use VM with no network and no credentials, which we destroy after the submission. That's an incident report, not a breach."

Key detail most candidates miss — test data never enters the sandbox as files. The harness outside the sandbox streams input via stdin and captures stdout; the expected output and checker stay outside. Otherwise user code can open("/tests/expected_17.txt") and print it.

Deep dive 2: The contest spike (8–10 min)

"Contests are on a calendar. Registration count tells me the peak a week in advance. I pre-warm the contest pool 30 minutes before start to 3× the forecast peak — roughly 300 submits/s × 2 CPU-s × 3 = ~1,800 cores — and keep it until 20 minutes after end for rejudges. Autoscaling handles only the unscheduled practice traffic."

"Fairness is decoupled from latency: penalty time is computed from ingress_ts, stamped at the submission service before the queue. If the queue lags 90 seconds, nobody's rank changes. The only user-visible cost is waiting for the verdict, and we cap that with lane priority."

Deep dive 3: Timing determinism (6–8 min)

"The verdict is CPU time from cgroup cpu.stat, not wall time. Each sandbox gets dedicated physical cores — I disable the SMT sibling for the contest pool. Wall time is a 3× backstop to kill sleep-forever programs. And I run the reference solution for 1% of problems every minute as a canary; if its CPU time drifts more than 5% from baseline, that host class is drained."

"Who pays: the pinning halves sandbox density — a 64-core host goes from ~120 sandboxes to ~48. For the contest pool I pay it. For the Run lane I don't — Run verdicts aren't ranked."

Phase 6: Wrap-Up (2–3 minutes)#

Synthesize, don't restate:

"The pipeline is a standard job queue. What makes it an online judge is three commitments: every submission is treated as an attacker and contained by a VM boundary; fairness is stamped at ingress so queue lag can't change rankings; and verdicts are measured in CPU time on pinned cores so they're reproducible."

The organizational closer:

"The failure I'd worry about most isn't the sandbox — it's wrong test data. That's owned by problem setters, not the platform team, and it causes mass rejudges. I'd make rejudge a first-class workflow with a diff report, and give the contest admin — not an engineer — the button, with the platform team owning the capacity it consumes."

🎯 Staff Move: End on the ownership seam between the platform (runtime, timing, capacity) and content (tests, checkers). That seam is where real incidents come from.

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

The online judge is a rare prompt where the "obvious" design is correct and also dangerously incomplete. Everyone gets queue + worker right. The level signal is whether you notice that (a) the workload is adversarial, (b) the peak is scheduled and therefore plannable, and (c) the product is a trustworthy measurement, not a computation. Interviewers use it to test threat modeling and capacity planning in one problem.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"A user submits the same C++ file twice, one minute apart, during a contest. The first gets Accepted in 940ms. The second gets TLE. Walk me through why — and what you'd have built so that this doesn't happen."

This question exposes whether the candidate distinguishes wall time from CPU time, understands noisy neighbors (shared L3 cache, memory bandwidth, SMT siblings), and treats verdict determinism as a measurable property with an owner. Candidates who've never operated a judge say "increase the timeout."

2. Problem Framing & Intent#

2.1 The Four Intents — Explained#

Practice judging → cost-first, eventually fast

• Constraint: steady ~115/s average with a diurnal peak around 300/s; margin matters because this is the free tier
• Strategy: shared autoscaled pool, weighted lane, can use cheaper isolation (gVisor) and denser packing
• Failure mode: verdict slows to 20–30s during peaks — annoying, not damaging
• Who pays for imperfection: free users (wait longer); support volume increases slightly

Contest judging → fairness-first

• Constraint: scheduled bimodal spike; ranking integrity is the product
• Strategy: dedicated, pinned, pre-warmed pool; ingress_ts for ordering; rejudge capacity reserved
• Failure mode: false TLE or backlog that users perceive as unfair → contest unrated
• Who pays for imperfection: the community's trust in rankings; the contest team's credibility; in the extreme, sponsors

Interactive Run → abuse-first

• Constraint: 3–5× the volume of Submit, near-zero value per call, easiest free compute on the internet
• Strategy: lowest priority, tight per-user CPU-seconds budgets (e.g., 60 CPU-s/hour for free, 600 for paid), shed first
• Failure mode: crypto mining, botnets farming free CPU
• Who pays for imperfection: the infrastructure bill; ultimately finance

Hiring assessment (B2B) → audit-first

• Constraint: contractual uptime (e.g., 99.9% during scheduled assessment windows), per-customer isolation, dispute resolution
• Strategy: separate tenant pool, full provenance per verdict (runtime image digest, host class, testset hash), 1–2 year retention
• Failure mode: candidate disputes a rejection; customer demands proof
• Who pays for imperfection: sales/legal; enterprise churn

🎯 Staff Move: "These four share one pipeline but not one policy. The architecture is lanes + pools; the intents differ in priority weight, isolation class, and retention. That's why I'd make lane and pool configurable, not hard-coded."

2.2 When NOT to Build This#

• You need to execute code for fewer than ~10K runs/day. Use a hosted sandbox service or an open-source judge (e.g., Judge0) on a managed VM. Building microVM orchestration for that volume is resume-driven.
• The code is trusted (internal CI, your own employees' notebooks). A standard container runner with resource limits is enough — the threat model changes everything.
• Languages are all JavaScript/WASM-compilable. V8 isolates or WASM runtimes give <5ms startup and dense packing; skip VMs entirely.
• Latency must be interactive (<100ms), e.g., a REPL-style playground. The judge pipeline (queue + batch tests) is the wrong shape — use a persistent per-session sandbox with a strict idle timeout instead.

2.3 What the Interviewer Leaves Underspecified#

Interviewers deliberately omit:

• Which languages. 3 languages vs 25 changes image management, warm-pool fragmentation, and seccomp tuning (JVM and Go need more syscalls).
• Custom checkers. Some problems accept multiple correct answers or floating-point tolerances; the checker is itself code — whose, and is it sandboxed?
• Interactive problems. Some judges run the user program against an interactor process over pipes — a two-process sandbox.
• Contest scale and frequency. One 30K-user contest per week vs a 300K-user global event changes whether you own reserved hardware.
• What users see on failure. Showing the failing test's input leaks hidden tests; hiding it frustrates practice users.

Staff engineers surface these. Senior engineers assume them away.

2.4 Precise Terminology#

3. The Five Fault Lines#

Each fault line below is a place where two reasonable engineers disagree. The Staff job is to pick, quantify, and name who absorbs the cost.

3.1 Fault Line 1: Isolation Strength vs Startup Latency#

The tension: Every layer of isolation costs startup time, runtime overhead, or density. The attacker only needs one gap.

Staff default: Container controls inside a Firecracker microVM for Submit and contest lanes. The warm pool hides the 125ms boot. For the Run lane, gVisor on shared hosts is acceptable because Run has no ranking impact — but it still gets no network and a strict CPU budget.

Reuse vs fresh sandbox: A microVM reused across submissions must be reset — a leaked background process, a file in /tmp, or a modified environment variable can alter the next user's verdict or leak their code. Staff default: one VM per submission, destroyed after. If density forces reuse, restore from a snapshot (Firecracker supports snapshot/restore) rather than "cleaning up."

When to deviate: Trusted code (internal assessments of employees) → containers alone. Extreme scale with only JS/Python → consider WASM (e.g., Pyodide) and accept the language-compatibility cost.

🎯 Staff Move: "I'm not choosing between container and VM — I'm stacking them. The container controls are for containment; the VM is for blast radius. The question I'd ask the security team is what our kernel CVE patch SLA is, because without a VM boundary that SLA is our sandbox's real security guarantee."

3.2 Fault Line 2: Warm Capacity vs Cost#

The tension: A pool sized for the contest peak is idle ~95% of the week. A pool sized for the average cannot catch a spike that starts in under 60 seconds.

Back-of-envelope:

Contest peak:     300 submits/s × 2 CPU-s = 600 cores busy
Headroom 3×:      1,800 cores  →  ~30 hosts of 64 cores (with SMT off: ~60 hosts)
Duration needed:  T-30min to T+20min after a 90-min contest ≈ 2.3 hours/week
Idle cost if permanent: 60 hosts × ~$3/hr × 168 h ≈ $30K/week
Scheduled cost:         60 hosts × ~$3/hr × 2.3 h ≈ $415/week (+ reservation premium)

Staff default: Calendar-driven pre-warm for contests (forecast = registrations × historical submits/user × peak factor), plus lane weights that borrow practice capacity during the contest window. Reactive autoscaling handles practice diurnal curves with a 20% floor of warm capacity per language.

Warm-pool fragmentation: With 20 languages, a warm pool per language fragments capacity. Mitigation: a language-agnostic base VM snapshot containing all runtimes (~2–4 GB image, cached on local NVMe), so any warm VM can run any language. Cost: bigger image, slower host bootstrap (minutes), but host bootstrap happens once per host, not per submission.

🎯 Staff Move: "Contests are the one spike in distributed systems that's printed on a calendar. Reacting to it with autoscaling is choosing to fail. I'd pre-warm from registrations and let practice absorb the squeeze, because a 20-second practice verdict costs nothing and a 5-minute contest verdict costs the contest."

3.3 Fault Line 3: Throughput vs Fairness Under Spike#

The tension: FIFO is simple and maximizes throughput. It also means a practice user's submission can sit ahead of a contestant's, and a single user with 50 rapid resubmits can occupy 50 slots.

Staff default: Weighted fair dispatch across lanes, plus per-user dedupe and fair share within the contest lane: at most 2 in-flight submissions per user per problem; identical source_hash resubmits return the cached verdict instantly.

The decoupling that makes this safe: Fairness in contests is about ordering and penalty, not verdict speed. Stamp ingress_ts at the submission service (before the queue). Penalty = ingress_ts − contest_start + 5–10 min per wrong attempt. Now queue lag affects user experience but not outcome. This is the single most important sentence in the contest deep dive.

When to deviate: In B2B assessments, each candidate has a private time limit; fairness is per-candidate latency, so a dedicated per-customer pool beats lane weighting.

🎯 Staff Move: "Queue lag is an experience problem, not a fairness problem — as long as the timestamp that decides rankings is taken before the queue. That lets me run the judge hot during the spike without anyone's rank depending on which worker picked them up."

3.4 Fault Line 4: Timing Determinism vs Packing Density#

The tension: Stable timing needs isolated hardware resources (dedicated cores, no SMT siblings, ideally no shared L3 thrash). Density needs sharing.

Staff default: CPU time on pinned physical cores with SMT disabled for the contest and B2B pools; CPU time on shared cores for practice; plus automatic re-run of any TLE within 10% of the limit before reporting it. Calibrate time limits per language (e.g., Python limits 3–5× C++), and publish them.

Canary detection: Every judge host continuously runs a small set of reference solutions (1% of capacity). Alert on judge.canary_cpu_ms drift > 5% from the host-class baseline; auto-drain the host. This catches thermal throttling, firmware changes, noisy co-tenants and kernel regressions — none of which show up in error rates.

🎯 Staff Move: "I'll pay 2× hardware for the contest pool because timing variance there changes rankings, and I'll refuse to pay it for practice where a borderline TLE just means the user resubmits. Determinism is a per-lane SLO, not a global one."

3.5 Fault Line 5: Judge Platform vs Content Ownership#

The tension: A wrong verdict can come from the platform (runtime version, timing, sandbox bug) or from content (wrong expected output, weak tests, buggy checker). Users can't tell the difference. Someone has to own the fix — and the rejudge.

Staff default: Split ownership with a testset contract:

• Every new testset version must pass: reference solution AC, at least one known-wrong solution fails, checker runs in the sandbox, max input size within limits.
• Rejudge is requested by content (contest admin), executed by the platform in a rejudge lane capped at 20% of capacity, with a diff report (how many verdicts flip, in which direction) reviewed before publishing.
• Verdicts are never overwritten; the "current" verdict points to the latest approved testset version.

🎯 Staff Move: "The rejudge button belongs to the contest admin, but its capacity belongs to me. I'd give them a dry-run diff — '1,240 verdicts flip from AC to WA' — before anything is published, because flipping 1,240 accepted verdicts is a community event, not a bug fix."

4. Failure Modes & Operational Reality#

4.1 Contest-Start Thundering Herd — Full Timeline#

Scenario: Weekly contest, 30K participants. The contest pool was sized from last month's registration, but a popular streamer announced the contest; 52K registered.

t=-30min:  Pre-warm to 3× forecast (forecast based on 30K). Capacity ≈ 900 submits/s burst.
t=0:       Contest opens. Problem pages served from CDN — fine.
t=+3min:   Easy problem submissions begin: 1,100 submits/s. Run clicks: 3,500/s.
t=+4min:   Contest lane lag p95 = 40s. Run lane lag = 4 min.
t=+5min:   Users see "Pending" for 40s and resubmit — 18% duplicate rate.
t=+6min:   Alert: queue_lag_p95{lane=contest} > 30s for 2 min → page.
t=+7min:   On-call shifts weights: contest 90 / practice 10 / run 0 (Run returns 503 + "Run disabled during peak").
t=+8min:   Source-hash dedupe enabled for resubmits (was off for Run lane).
t=+12min:  Reactive capacity from reserved warm hosts arrives. Lag p95 = 8s.
t=+85min:  Second peak (last-minute submissions): absorbed, lag p95 = 12s.
t=+95min:  Leaderboard final at t+97min. Rankings unaffected — ingress_ts ordering.

Detection: queue_lag_p95{lane}, submissions_per_sec{lane}, duplicate_submit_ratio, warm_pool_available{language}.

Mitigation: Shift lane weights (pre-approved playbook), disable Run for non-contest users, enable dedupe, pull reserved capacity.

Prevention: Forecast from live registrations up to T-1h, not last month; auto-add capacity when registrations exceed forecast by 25%; client-side resubmit debounce (disable Submit button until verdict or 30s).

Owner: Platform on-call executes; contest team owns the forecast input.

4.2 Noisy Neighbor → False TLE → Rejudge Storm#

Scenario: A new host class (different CPU generation) was added to the contest pool to handle growth. Its per-core performance is 12% lower for memory-bound code. Nobody recalibrated time limits.

t=0:       New host class takes 30% of contest pool.
t=+20min:  Forum posts: "same code AC then TLE".
t=+25min:  judge.canary_cpu_ms on new class = +14% vs baseline. No alert (threshold was on errors, not drift).
t=+60min:  Contest ends. 2.1% of contest submissions TLE within 15% of limit.
t=+2h:     Contest team requests rejudge of all TLE verdicts. 9K submissions.
t=+2h10m:  Rejudge runs in practice lane, uncapped. Practice lag climbs to 11 min.
t=+4h:     Rejudge complete. 640 verdicts flip TLE → AC. Rankings recomputed. Public apology.

Detection: judge.canary_cpu_ms drift by host class (should have alerted at 5%), verdict_flip_rate_on_rerun, tle_near_limit_ratio (TLEs within 10% of the limit — normally ~0.3%).

Mitigation: Drain the host class from the contest pool; rejudge in a capped rejudge lane.

Prevention: Host classes are admitted to the contest pool only after a calibration run of the full reference suite; per-class time limit multipliers; borderline TLE auto-rerun on a baseline-class host.

Owner: Platform team (host qualification); contest team (communication and rejudge approval).

4.3 Sandbox Escape via Kernel CVE#

Scenario: A public Linux kernel privilege-escalation CVE is disclosed. The exploit uses a syscall on your seccomp allowlist.

t=0:       CVE published with proof-of-concept.
t=+6h:     First submissions containing the PoC appear (security researchers and attackers both read the feed).
t=+6h:     Container-only judge: exploit gains root on host → reads other sandboxes' source, host credentials.
           MicroVM judge: exploit gains root in the guest VM → no network, no creds, VM destroyed after run.
t=+8h:     Detection: sandbox_violation_total spike (seccomp kills, unexpected uid 0 in guest).
t=+24h:    Guest and host kernels patched and rolled via rolling drain.

Detection: sandbox.seccomp_kill_total, sandbox.guest_uid0_events, anomaly on submission_source_signature (known PoC hashes), host integrity monitoring.

Mitigation: Block the syscall in seccomp immediately (config push, minutes) if runtimes tolerate it; emergency kernel patch; quarantine suspicious submissions.

Prevention: VM boundary, no credentials on judge hosts (pull-only, scoped tokens), guest kernel patch SLA ≤ 72h for critical CVEs, source scanning for known PoC signatures, bug bounty including sandbox escapes.

Owner: Security (threat intel, SLA) + platform (patch rollout). The blast radius is determined by an architecture decision made months earlier.

4.4 Wrong Test Data — Silent Mass Misjudgment#

Scenario: A problem setter uploads a testset where case 37's expected output was generated by a buggy reference solution. Correct submissions get Wrong Answer.

Detection signals: problem_ac_rate drops sharply vs difficulty-predicted rate; wa_concentration_by_case (80% of WAs fail on the same case); user reports. The concentration metric is the strongest signal — real WAs spread across cases.

Mitigation: Freeze the problem (stop judging new submissions, return "Pending review"), fix testset, rejudge affected submissions via the rejudge workflow with a diff report.

Prevention: Testset contract (§3.5): multiple independent reference solutions (in two languages) must agree; known-wrong solutions must fail; automated alert when one case accounts for > 50% of WAs within the first 200 submissions.

Owner: Content team owns the testset; platform owns the concentration alert and rejudge capacity.

4.5 Zombie Sandboxes — Capacity Silently Shrinks#

Scenario: A runtime bug means some sandboxes are not destroyed when the harness times out (e.g., the VM process hangs on teardown). Each leaked VM holds 256 MB and 1 pinned core.

Detection: host.allocated_sandboxes − host.active_executions (should be ~0), warm_pool_available trending down without load growth, per-host capacity vs expected.

Mitigation: Reaper process kills any VM older than max_wall × 2; hosts whose leak count exceeds 5 are drained and recycled.

Prevention: Hosts are cattle: recycle every 24h regardless; teardown is part of the canary suite.

Owner: Platform team. This failure never pages on its own — it shows up as "the contest pool was smaller than we thought."

4.6 Free Compute Farming via Run#

Scenario: A botnet of 800 free accounts uses the Run endpoint to mine cryptocurrency, each run using the full 10s wall limit with multi-threaded hashing.

Detection: run_cpu_seconds_per_account distribution (bots hug the limit), run_output_entropy, identical source hashes across many accounts, sign-up velocity by IP/ASN.

Mitigation: Per-account CPU-seconds budget (60 CPU-s/hour free), per-IP/ASN budgets, cap threads (pids.max, 1 CPU per Run sandbox), block known mining signatures.

Prevention: Run is 1-core, 5s, lowest priority, shed first; new accounts get smaller budgets until they build history; CAPTCHA on anomalous sign-up bursts.

Owner: Trust & safety (account policy) + platform (quotas). Finance is the silent victim — the bill arrives a month later.

4.7 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Deep knowledge of Linux namespaces ≠ sandbox design. Listing all seven namespaces without stating the trust boundary is trivia.
• Kubernetes fluency ≠ capacity planning. "HPA on queue depth" is still reactive scaling.
• Mentioning Firecracker ≠ understanding why. The signal is explaining what the VM boundary changes about blast radius, and what it costs.
• Elaborate leaderboard design ≠ contest fairness. A perfect sorted set is useless if the timestamp feeding it came from the judge instead of ingress.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Problem browsing, tags, discussions, editorial pages. CDN-cached reads — say so in one sentence.
• The code editor. Client-side; irrelevant.
• Plagiarism detection internals. Mention it as an offline post-contest job (token-based similarity, MOSS-style); don't design it.
• Leaderboard data structure. A sorted set keyed by (solved desc, penalty asc) is enough; link to Leaderboard if pushed.

6.4 Follow-Up Questions to Expect#

1. "Why not run all test cases in parallel across workers?" — (Fan-out increases VM count 50–100× per submission; latency gain is small because most submissions fail early or finish in 1–3s total. Parallelize only for B2B problems with long tests.)
2. "How do you handle a submission that compiles for 60 seconds?" — (Separate compile limit, e.g., 10s CPU / 30s wall; template-metaprogramming bombs are real.)
3. "What does the client do while waiting?" — (Poll with backoff, or SSE for contest pages; show queue position to reduce resubmits.)
4. "How do you prevent hard-coding answers to known test cases?" — (Hidden tests, randomized large cases, post-contest review; it's a content problem, not a runtime one.)
5. "How would you judge interactive problems?" — (Two sandboxed processes connected by pipes, interactor outside the user's VM or in a separate one; measure only the user's CPU.)
6. "What's the cost per submission?" — (~2 CPU-s at ~$0.04/core-hour ≈ $0.00002; the expensive part is idle warm capacity, not execution.)
7. "How do you roll out a new compiler version?" — (New runtime image = new "runtime version" on verdicts; shadow-judge 1% of submissions on both versions and diff verdicts before switching; never switch during a contest.)

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design LeetCode."

Drill 2: The Sandbox#

Prompt: "How do you make sure user code can't harm the system?"

Drill 3: Make the Spike Concrete#

Prompt: "A contest with 30K participants starts at 8:00. Walk me through capacity."

Drill 4: The Flaky Verdict#

Prompt: "Users say the same code sometimes gets AC and sometimes TLE."

Drill 5: Wrong Test Data During a Contest#

Prompt: "Twenty minutes into a contest, you realize problem C's test 37 has a wrong expected output."

Drill 6: Multi-Language Support#

Prompt: "We support 5 languages. Product wants 30."

Drill 7: The Run Button Is Being Abused#

Prompt: "Our compute bill doubled. Run is 80% of CPU."

Drill 8: Build vs Buy#

Prompt: "Why not just use a hosted code execution API or run Judge0?"

Drill 9: Multi-Region#

Prompt: "We're launching in India and Brazil. Do we need judges there?"

Drill 10: Changing the Compiler Without an Outage#

Prompt: "We need to upgrade GCC. How do you roll it out?"

8. Deep Dive Scenarios#

Deep Dive 1: The Contest That Melted#

Context: Saturday's contest had 48K participants — 60% more than forecast. Verdict latency hit 9 minutes at t+6min, users resubmitted, and the contest team is asking whether to declare it unrated. The on-call escalates to you at t+10min.

Questions to Surface First:

• Are rankings actually affected, or only experience? (Is penalty time from ingress_ts or verdict time?)
• Which lane is lagging — contest, or is the Run lane stealing capacity?
• What fraction of the queue is duplicate resubmits?
• How much reserved capacity can arrive in the next 5 minutes, and at what cost?

Typical L5 Approach: Scales out the worker deployment, raises the autoscaler max, and watches queue depth fall over 15 minutes. Correct and slow. Doesn't address resubmit amplification or tell the contest team whether rankings are safe.

Staff Approach: Executes the pre-approved spike playbook: shift weights to contest 90 / practice 10 / run 0, enable source-hash dedupe, show queue position in the UI to stop resubmits, pull reserved hosts. Confirms rankings use ingress_ts, and tells the contest team: "Experience is degraded, outcomes are not — no need to unrate."

Principal Approach: Fixes the forecast input, not the incident: capacity is driven by live registrations with an automatic top-up at +25%, and contests above a size threshold require a capacity review 72h ahead. Negotiates a scheduled-capacity reservation for the weekly calendar so the top-up is guaranteed, and writes the unrated-contest criteria so the decision is policy, not mood.

Key Takeaway: "If fairness is stamped at ingress, a slow judge is an experience incident, not an integrity incident."

What clears the Staff bar:

• Asks whether rankings are affected before scaling anything
• Uses a pre-approved playbook rather than improvising
• Attacks the amplifier (resubmits), not just the capacity

Deep Dive 2: Silent Timing Drift#

Context: A power-user publishes a blog post showing that the same Rust solution runs 18% slower on your judge on weekends. There are no alerts. Your error rates are flat.

Questions to Surface First:

• What's different about weekend capacity — host class, co-tenancy, spot instances?
• Are canaries running per host class, and what are their thresholds?
• How many borderline TLEs were issued on those hosts?

Typical L5 Approach: Increases time limits by 20% across the board. That hides the drift and makes weak solutions pass on fast hosts.

Staff Approach: Finds that weekend autoscaling uses a cheaper, older instance family. Adds per-class canaries with 5% drift alerts, excludes unqualified classes from the contest pool, and reruns borderline TLEs from the affected window.

Principal Approach: Treats "verdict determinism" as an SLO with an owner and a budget. Instance-family choices that affect verdicts now require qualification; procurement cost savings are weighed against the determinism SLO explicitly, not decided by an autoscaler config.

Key Takeaway: "Timing drift never shows up in error rates. If you don't canary the measurement, you don't know your measurement."

What clears the Staff bar:

• Refuses the blanket limit increase
• Builds detection for a failure that emits no errors

Deep Dive 3: Enterprise Assessment Customer Onboarding#

Context: A large company wants to use your judge for 200K candidate assessments per year, with their own problems, custom checkers, SSO, 99.9% availability during scheduled windows, and verdict reproducibility for two years.

Questions to Surface First:

• Do their checkers run in our sandbox? (They're untrusted code too.)
• What are the concurrency peaks — e.g., university hiring days?
• Data residency? Retention? Who handles candidate disputes?

Typical L5 Approach: Adds a tenant_id column, lets them upload problems, runs everything on the shared pool.

Staff Approach: Tenant-dedicated pool (or reserved share) so a public contest can't delay their assessment; checkers sandboxed with the same controls; verdict provenance (runtime digest, host class, testset hash) stored for 2 years; old runtimes retained for reproduction.

Principal Approach: Decides whether B2B is a product line with its own SLA, on-call and pricing, or a side feature. If it's a product line, it gets a separate cell (blast-radius isolation from public contests), a contract review for liability on disputed verdicts, and a runtime deprecation policy written into the contract.

Key Takeaway: "Enterprise customers don't buy execution — they buy a verdict they can defend in a dispute."

What clears the Staff bar:

• Notices the custom checker is untrusted code
• Designs for reproducibility, not just correctness

Deep Dive 4: Post-Mortem — Test Data Exfiltration#

Context: Hidden test cases for 40 premium problems appeared on GitHub. Investigation shows users printed them via a side channel: the sandbox gave different error messages depending on input content, and a script submitted thousands of probes.

Questions to Surface First:

• What does the verdict reveal? (Failing case index, input on failure, stderr?)
• Was the testset readable from inside the sandbox at any point?
• How many accounts, and what's the probe rate?

Typical L5 Approach: Rotates the tests and blocks the accounts.

Staff Approach: Classifies it as an information-leak design flaw: verdict detail + unlimited submissions = oracle. Limits what's returned (first failing case index only for hidden tests; no stderr for hidden tests), adds per-problem submission budgets, detects probe patterns (many submits with minimal source diffs, WA on incrementing case index), and regenerates testsets with randomized large cases.

Principal Approach: Recognizes this as a product-policy question with security consequences: how much feedback do we give vs how much we leak. Sets the policy jointly with product (practice problems get rich feedback on sample tests only), and adds leak detection (monitoring public repositories for testset fingerprints) as a standing capability.

Key Takeaway: "A judge with unlimited attempts and rich feedback is an oracle. Feedback is a security surface."

What clears the Staff bar:

• Identifies the oracle pattern, not just the leak
• Pairs technical fixes with a feedback policy

Deep Dive 5: Multi-Region Expansion for a Global Contest#

Context: Leadership wants a 250K-participant global contest with users on every continent. Current judge runs in one US region.

Questions to Surface First:

• Where is ingress_ts taken, and does RTT affect tie-breaks?
• Can one region absorb ~2,500 submits/s peak?
• What happens if the judge region fails mid-contest?

Typical L5 Approach: Deploys judges in three regions and routes users to the nearest.

Staff Approach: Regional ingress stamps ingress_ts (clocks disciplined to < 10ms via NTP/PTP), submissions replicate to a central queue; judging stays in two regions (active + warm standby sized for 100%). Test data prefetched to both. Lane weights and capacity from registrations by region.

Principal Approach: Weighs the one-off cost (~2 regions × peak pool × a few hours) against the reputational cost of a failed flagship event; runs a game day with a synthetic 250K-user contest two weeks prior; decides whether a staggered start (problems released by region) is a product answer that removes the infrastructure problem.

Key Takeaway: "Latency matters where you take the timestamp, not where you run the code."

What clears the Staff bar:

• Separates fairness latency from execution latency
• Plans DR as a drill, not a diagram

9. Level Expectations Summary#

After studying this case study, you should be able to:

• State the threat model in the first 3 minutes and design a layered sandbox with a VM boundary
• Explain why test data must never enter the sandbox
• Size a contest pool from registrations and explain why reactive autoscaling fails
• Decouple ranking fairness from queue latency with ingress timestamps
• Explain CPU time vs wall time, noisy neighbors, SMT, and canary-based drift detection
• Design rejudge as an idempotent, versioned, diff-reviewed workflow
• Distinguish Run and Submit economically and design abuse budgets in CPU-seconds
• Articulate the content-vs-platform ownership seam and the org policy for unrated contests

The Bar for This Question#

Mid-level (L4): API, queue, worker in Docker, database. Mentions timeout and no network. Needs prompting to discuss spikes.

Senior (L5): Adds autoscaling, retries, per-user rate limits, maybe multiple languages. Correct system for a normal day. Security story is "containers"; timing is wall-clock; rejudge is not considered.

Staff+ (L6): Treats code as hostile, stacks containment inside a VM boundary, keeps test data outside, pre-warms for scheduled spikes, stamps fairness at ingress, measures CPU time on pinned cores with canaries, and owns the wrong-verdict path through versioned testsets and rejudge. Names who waits, who pays, and who decides. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Containers Are Not a Security Boundary for Public Judges"#

The Staff position: Containers are a containment mechanism. The security boundary is the VM (or gVisor). Pay the ~125ms and ~10–15% density.

Why this matters in interviews: Saying "Docker is secure enough" is the single fastest way to cap your level on this problem.

10.2 "Queue Latency Doesn't Matter for Contest Fairness"#

The Staff position: Stamp at ingress and spend your money on determinism, not on shaving seconds off a verdict nobody is ranked by.

Why this matters in interviews: It shows you can find the invariant that makes a scary problem cheap.

10.3 "Most Judge Incidents Are Content Bugs, Not Platform Bugs"#

The Staff position: Build the problem-setter validation pipeline (multiple reference solutions, known-wrong solutions, stress generators) before optimizing the runtime further.

Why this matters in interviews: Naming the non-obvious top source of incidents shows operational experience.

10.4 "The Run Button Should Cost Users Something"#

The Staff position: Budget Run in CPU-seconds per user, visible in the UI, with larger budgets for paid tiers. Free-and-unlimited is a policy choice with a monthly bill.

Why this matters in interviews: It turns a rate-limit answer into an economics answer.

10.5 "Parallelizing Test Cases Is Usually a Mistake"#

The Staff position: Run tests sequentially with early exit in one sandbox; parallelize only for problems with long individual tests, and only outside contests.

Why this matters in interviews: It shows you evaluate an "obvious optimization" against the spike it would make worse.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer designs a judge. A Principal engineer notices that "execute untrusted code safely" is about to be requested by the assessment product, the AI tutor that runs model-generated code, a notebook feature, and a plugin system — and that four teams building four sandboxes means four security postures, four CVE response times, and four cost curves. The L7 question is not "which sandbox" but "which sandbox platform, with what contract, owned by whom, and what do we refuse to support."

The Org-Level Fault Line#

One execution platform vs per-product sandboxes. Centralizing gives one security posture, one patch SLA, shared warm capacity and shared calendar-driven scheduling. It also creates a bottleneck team and forces the judge's determinism requirements (pinned cores) onto products that don't need them. The L7 answer: centralize the isolation primitive and the security contract (image, seccomp profiles, VM boundary, patch SLA); let products own scheduling policy and pools (lanes, determinism class, retention).

Cost Model#

Assumptions: $0.04 per core-hour (bare metal amortized), ~2 CPU-s per submission, engineers at $250K fully loaded.

The dominant cost at medium scale is idle warm capacity and pinned-core density loss, not execution. The dominant cost at small scale is engineers — which is why buying wins there.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Untrusted Code Execution Standard (v1)

Scope: Any service that executes code not authored by our employees — judge, assessments, AI-generated code, plugins.

MUST:

• Execute inside the platform sandbox runtime (VM boundary + seccomp + cgroups v2 + no default network).
• Never mount secrets, credentials or evaluation data inside the sandbox.
• Record runtime image digest and host class on every result that is shown to a user.
• Patch guest and host kernels within 72h of a critical CVE.
• Enforce per-principal CPU-second budgets.

SHOULD:

• Use calendar-driven capacity for scheduled events.
• Declare a determinism class (strict / standard / best-effort) per workload.

Exceptions: Filed with the platform team and security; time-boxed to 90 days; reviewed quarterly.

Success metrics: Zero host compromises; ≤ 72h critical patch time; false-TLE rate < 0.1% on strict class; ≤ 1 execution platform per company.

What I'd Tell the VP#

Our code execution is the core of the product and our biggest security exposure: we run millions of programs a day written by strangers. Today it's safe enough, but contests stall when traffic beats our forecast, and timing inconsistencies occasionally make results look unfair — which is what our community notices. I'm proposing we invest roughly five engineers for two quarters to make contest capacity scheduled rather than reactive, make verdicts reproducible, and turn the sandbox into a shared platform that the assessment product and AI tutor will also use. That avoids three teams each building their own sandbox, and it converts our largest reputational risk into a measured SLO.

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll use Firecracker" — correct, but doesn't address who owns patching or how the next team reuses it.
• "Pre-warm for contests" — correct, but doesn't price the idle pool or propose reserved capacity.
• "Rejudge with a diff report" — correct, but doesn't set the policy for when a contest is unrated, or who decides.

🧭 Principal Move: "The judge is our first code-execution product, not our last. I'd spend the extra quarter making the sandbox a platform with a written contract, because the alternative is four sandboxes with four different answers to the next kernel CVE."

Appendices

Appendix A: Sandbox Mechanics in Depth#

A.1 The Execution Harness#

The harness runs outside the user's sandbox and owns every trusted artifact.

judge(submission, testset):
    vm = warm_pool.acquire(runtime=submission.language)        # ~10ms
    vm.put_file("/work/main.src", submission.source, ro=False)
    r = vm.exec(compile_cmd, cpu_limit=10s, wall=30s, mem=1GB)
    if r.status != OK: return verdict(COMPILE_ERROR, r.stderr[:4KB])
    for i, case in enumerate(testset.cases):                    # streamed from local NVMe cache
        r = vm.exec(run_cmd, stdin=case.input,
                    cpu_limit=problem.tl, wall=3*problem.tl,
                    mem=problem.ml, pids=64, stdout_cap=64KB)
        if r.cpu_ms > problem.tl:      return verdict(TLE, i, maybe_rerun=near_limit(r))
        if r.oom:                       return verdict(MLE, i)
        if r.exit != 0:                 return verdict(RE, i)
        if not checker(case, r.stdout): return verdict(WA, i)   # checker runs outside, or in its own sandbox
    return verdict(AC, max_cpu, max_mem)
    finally: vm.destroy()                                        # never reuse without snapshot restore

A.2 seccomp Allowlist Strategy#

• Start from an allowlist (~60 syscalls: read, write, mmap, brk, exit_group, futex, clock_gettime, …).
• Per-language additions (JVM: clone with thread flags, sched_yield, madvise; Go: signals).
• Default action KILL_PROCESS and emit sandbox.seccomp_kill_total{syscall} — kills on odd syscalls are attack telemetry.

A.3 Why Destroy, Not Clean#

A "cleaned" sandbox must prove the absence of: background processes, modified files, changed env, kernel state (e.g., keyrings), and page cache side channels. Destroy-and-reacquire from a warm pool (or snapshot restore at ~tens of ms) is cheaper than proving cleanliness.

Appendix B: Data Model#

Sharding: submissions and verdicts by submission_id hash; user history via a secondary index on (user_id, problem_id, ingress_ts). See Sharding.

Appendix C: Dispatch and Queueing#

C.1 Weighted Fair Dispatch#

weights = contest_active ? {contest:70, practice:25, run:5} : {practice:80, run:20}
on worker_slot_free:
    lane = weighted_choice(nonempty_lanes, weights)       # deficit round robin
    job  = lane.pop_fair_by_user()                        # max 2 in-flight per (user, problem)
    if cache.has(job.source_hash, testset_version): return cached verdict

C.2 Quick Comparison#

Durability rule: write the submission row before enqueue; a reconciler re-enqueues QUEUED rows older than 60s. See Message Queues.

Appendix D: Client Contract#

• POST /submissions returns 202 with submission_id and queue_position.
• Poll GET /submissions/{id} with backoff 500ms → 1s → 2s (cap 2s); or SSE on contest pages (Real-time Updates).
• Submit button disabled until verdict or 30s; identical source within 60s returns the existing submission_id.
• 429 with Retry-After when user budget exhausted; 503 with a reason for Run during peak shedding.

Appendix E: Observability#

E.1 Core Metrics#

queue_lag_p95{lane}                  verdict_latency_p95{lane}
submissions_per_sec{lane}            duplicate_submit_ratio
warm_pool_available{language}        host.allocated_sandboxes - host.active_executions
judge.canary_cpu_ms{host_class}      tle_near_limit_ratio{host_class}
wa_concentration_by_case{problem}    sandbox.seccomp_kill_total{syscall}
run_cpu_seconds_per_account (p99)    rejudge_lane_utilization

E.2 Critical Alerts#

E.3 Debugging the Silent Failure#

Timing drift and zombie sandboxes emit no errors. The only defense is synthetic workload: canary reference solutions and teardown checks that run continuously and compare against baselines.

Appendix F: Scale Evolution#

What You Don't Build on Day One#

• Regional judges (ingress timestamping is enough until residency forces it)
• Parallel test execution
• Custom hypervisors or kernels
• Plagiarism detection in the hot path (it's an offline post-contest job)

Appendix G: Multi-Tenancy and Cost#

• Chargeback unit: CPU-seconds by lane and product; warm-floor cost allocated by reserved share.
• Fairness within Run: per-account and per-ASN CPU-second budgets; trust tiers by account age and history.
• B2B isolation: reserved share or dedicated cell; a public contest must never delay a paid assessment window.
• Tradeoff summary: strict determinism costs ~2× hardware; VM boundary ~10–15% density; calendar pre-warm costs hours per week instead of a permanently idle peak pool. Spend where verdicts are ranked or contracted, save everywhere else.