Technologies referenced in this case study: Kafka · Cassandra · Redis · Elasticsearch · DynamoDB

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once, then return to individual sections for targeted review.

Without per-host politeness:
t=0:      New crawler launched, 2,000 fetchers, frontier ordered by PageRank only
t=+5min:  Top 50 hosts each receive 300-800 req/s from our IP range
t=+12min: A mid-size news site's origin falls over; their on-call traces it to us
t=+40min: Our /16 is blocked by two CDNs; 18% of fetches now return 403
t=+1day:  Abuse complaints to our cloud provider; legal gets an email
t=+3days: Freshness for the top 10K hosts is WORSE than before launch

With host-partitioned politeness:
t=0:      Same launch, same 2,000 fetchers
t=+5min:  Each host capped at 1 concurrent connection, >= 1s between fetches
t=+5min:  Fetchers stay busy by interleaving ~500K distinct hosts
t=+1hr:   Adaptive delay backs off hosts whose latency or 5xx rate rises
t=+1day:  Zero complaints; aggregate throughput identical; ban rate < 0.1%

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What This Interview Actually Tests#

A web crawler is not a graph traversal question. Everyone can write BFS.

It is a scheduling-under-external-constraints question that tests:

• Whether you clarify what the corpus is for before sizing anything
• Whether you treat other people's servers as a constrained, shared resource you do not own
• Whether you can defend against adversarial and pathological input at 10⁹ scale
• Whether you can own freshness as an SLO rather than a hope

The key insight: The throughput of a crawler is bounded not by your fetchers but by politeness — the number of distinct hosts you can interleave. Staff engineers design the frontier around the host, not the URL, and they name who gets hurt when the crawler misbehaves: the site operator first, your IP reputation second, your index freshness third.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll assume a general search-index crawler: roughly 1B pages kept fresh on a monthly cycle, with the top 10M refreshed daily and news hosts refreshed hourly. That puts the hard design in two places — the host-partitioned frontier that enforces politeness, and the recrawl scheduler that decides which 40M fetches a day are worth spending. Archive and change-monitoring crawls reuse the fetch layer but need a different scheduler."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Citing specific, publicly documented crawlers shows you have studied operational reality, not just a textbook BFS.

Mercator (Compaq/DEC research) — The Front-Queue/Back-Queue Frontier#

Heydon and Najork's Mercator paper (1999) introduced the frontier design most production crawlers still resemble: front queues encode priority, back queues hold one host each, and a heap keyed by "earliest next fetch time per host" decides which back queue a fetcher thread serves. It also documented a URL-seen test with an in-memory cache in front of a disk-resident set, and content fingerprinting to skip duplicate documents.

Staff insight: Mercator's contribution was not speed — it was making politeness a structural property of the queue rather than a check in the fetcher. That is the sentence to say in an interview.

Google — Near-Duplicate Detection with SimHash#

Manku, Jain, and Das Sarma (WWW 2007) described using 64-bit SimHash fingerprints over a multi-billion-page crawl, treating pages within Hamming distance 3 as near-duplicates, and building permuted fingerprint tables so the lookup stays fast. Google also documents that Googlebot caches robots.txt for up to 24 hours and backs off when a site returns 5xx or 429.

Staff insight: Near-dup detection is a budget mechanism, not a storage-saving trick. Every near-dup you fetch is a fetch you didn't spend on fresh content — dedup protects freshness.

Common Crawl — The Open Bulk Corpus#

Common Crawl publishes monthly-ish crawls of a few billion pages each as WARC files on public cloud object storage, respecting robots.txt with an identifiable CCBot user agent. It is the canonical example of the bulk corpus intent: little recrawl scheduling, heavy emphasis on coverage, provenance, and a stable archival format that downstream consumers (including many ML training pipelines) re-parse independently.

Staff insight: Storing raw bytes in a standard format decouples the crawl from every consumer's parser. When the "consumer" becomes 30 teams, that decoupling is the platform.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: The frontier is partitioned by host so a single worker owns each host's queue, robots.txt, and crawl delay — politeness needs no coordination. Fetchers write raw bytes to object storage before parsing, so a parser bug is a reprocessing job, not a month of refetching. The URL-seen store closes the loop: only never-seen URLs re-enter the frontier. The recrawl scheduler is the only component that decides which known pages to spend budget on — it is where freshness is won or lost.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 20 minutes drawing seed → queue → fetch → parse → store and computing storage for 1B pages, then run out of time before politeness, freshness, or traps come up. Compress the pipeline to 5 minutes. The level is decided in the frontier and the scheduler.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional scope in one sentence:

"We're building a crawler that discovers and fetches web pages, extracts links, and hands content to a search indexer — continuously, not as a one-shot job."

Then spend your time on the non-functional requirements that actually shape the design:

"The number I'll design around is the recrawl budget: at ~40M fetches a day, which 40M pages are worth fetching? That's where most of the interesting tradeoffs live."

Phase 2: Core Entities & API (1–2 minutes)#

Keep this tight. The entities that matter:

Host       { host, ip_set, robots_rules, robots_fetched_at, crawl_delay_ms,
             budget_remaining_today, health_score, owner_partition }
UrlRecord  { url_hash (64-bit), canonical_url, host, first_seen, last_fetched,
             last_changed, change_rate_est, importance, etag, last_modified,
             content_hash, simhash, status }
FetchTask  { url_hash, priority_band, not_before, attempt }
FetchResult{ url_hash, http_status, headers, body_ref (WARC offset), fetched_at }

Internal APIs — not public HTTP, but name them:

frontier.enqueue(urls[], source, priority_hint)
frontier.lease(worker_id, max_hosts) -> [HostBatch]
fetch.result(FetchResult)                    // async, to Kafka
policy.check(host) -> {allowed, delay_ms, budget}
scheduler.recrawl_candidates(partition, n)   // daily or hourly

"The key identity decision: canonical URL after normalization, hashed to 64 bits. Two URLs that normalize the same are the same page for dedup, even if the server disagrees."

Phase 3: High-Level Architecture (≤5 minutes)#

Staff candidates spend under 5 minutes here. Draw the loop, name the partitioning key, move on.

Say it in four sentences:

1. "Frontier is partitioned by hash(host) — each partition owner holds the host's queue, robots.txt, crawl delay, and budget."
2. "Fetchers lease batches of URLs per host, fetch asynchronously with a local DNS cache, and write raw bytes to object storage before anything parses them."
3. "Parsing, link extraction, and dedup are downstream stream processors; new URLs go through the URL-seen store and re-enter the frontier on their host's partition."
4. "A separate recrawl scheduler injects known URLs based on estimated change rate and importance — that's the freshness engine."

Phase 4: Transition to Depth (1 minute)#

The sentence that steers the interviewer:

"The pipeline is the easy part. The three places I'd like to go deep are: how the frontier enforces politeness without a distributed lock, how the recrawl scheduler spends a fixed budget to maximize freshness, and how we defend against spider traps and duplicate content that silently eat that budget. Which is most interesting to you?"

If they don't pick: go frontier → traps → freshness. The frontier is the architectural core; traps show operational ownership; freshness shows product judgment.

Phase 5: Deep Dives (25–30 minutes)#

Deep dive 1 — The host-partitioned frontier (8–10 min).

• Partition key hash(registered_domain) or hash(host). Registered domain (eTLD+1, via the Public Suffix List) is safer: a.example.com and b.example.com often share one origin server.
• Within a partition, each active host has a back queue. A min-heap keyed on next_fetch_at picks the next ready host. After a fetch, next_fetch_at = now + max(crawl_delay, k × last_response_time) with k ≈ 10 — slow servers automatically get more spacing.
• Front queues (priority bands 0–7) feed back queues; a biased random selector favors high bands without starving low ones.
• Fetchers are stateless and lease HostBatches (e.g., up to 20 URLs for one host, served sequentially on one keep-alive connection).
• Rebalancing: partitions map to workers via consistent hashing; a worker failure moves its hosts, and the new owner must re-fetch robots.txt state from the shared cache before fetching.

"The throughput math is: fetches/s ≤ active_hosts / avg_delay. With 1s delay and 500K active hosts, the ceiling is 500K/s — far above our 385/s need. So the constraint isn't the head of the web; it's that the top 1,000 hosts hold maybe 10% of the URLs we care about and each can only take ~86K fetches a day."

Deep dive 2 — Spider traps and budget defense (6–8 min).

• Per-host daily URL budget (default 10K–100K, raised for known-large hosts like Wikipedia via an allowlist with owner).
• URL normalization: lowercase scheme/host, drop default ports, sort query params, strip known session params (sid, jsessionid, utm_*), remove fragments.
• Trap heuristics: path depth > 16, repeated path segments (/a/b/a/b/a/b), URL length > 2 KB, per-host new-URL rate far above the host's historical baseline, near-dup ratio > 80% within a host.
• Suspected hosts drop to the lowest priority band and go to a review queue — never silently blocked, because false positives are how you lose a real site.

Deep dive 3 — Freshness scheduling (8–10 min).

• Estimate per-URL change rate λ from fetch history (Poisson model: P(changed within t) = 1 − e^(−λt)).
• Priority for recrawl = importance × P(changed since last fetch). Importance comes from link-graph signals and query logs — owned by the ranking team.
• Use conditional GET (If-None-Match, If-Modified-Since): a 304 costs ~0.5 KB instead of ~100 KB. Sitemaps' lastmod and RSS/Atom feeds give free change hints for news hosts.
• Hard floors: top tier never older than 24h; everything recrawled at least every 90 days so dead links get detected.

Phase 6: Wrap-Up (2–3 minutes)#

Close with constraints, evolution, and what you would build later:

"To summarize: host-partitioned frontier so politeness is structural, raw bytes stored before parsing so we can reprocess without refetching, three-layer dedup, and a recrawl scheduler that spends ~40M daily fetches on importance × change probability. What I'd build next: JavaScript rendering for the ~5% of high-value pages that need it, a crawl-as-a-platform API so ML and trust & safety teams stop running their own crawlers against the same hosts, and a freshness dashboard per tier that search leadership signs off on."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Crawling is one of the few classic prompts where the system's main constraint is imposed by parties you don't control. Rate limiters protect your backend; caches protect your database. A crawler must protect other people's servers, obey rules those people publish in a 30-year-old text format, and survive adversarial content they serve back. That makes it a clean test of whether a candidate designs for an ecosystem or just for a throughput number.

It also tests budget allocation. A crawler never has enough fetches. The Staff skill is deciding what not to fetch — and making sure that decision is observable and owned.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"Which fetch are we not doing today, and who decided that?"

Every crawler is budget-constrained: by politeness, by bandwidth, by storage, by downstream indexing capacity. The question forces the design to expose its allocation policy. If the answer is "whatever the queue happens to pop next," nobody owns freshness. If the answer is "the scheduler deprioritized 12M tail pages whose estimated change probability is under 2%, per the policy search ranking signed off on," you have a system someone can run.

🎯 Staff Move: "I want to make the skip decisions explicit. Every day the scheduler emits a report — fetches allocated by tier, fetches denied by budget, hosts throttled by politeness — so when someone asks why a page is stale, we can answer from data instead of guessing."

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Intent 1 — Search-index freshness crawl. Continuous. The corpus is mostly known; the job is keeping it current and discovering the new 1–3% that appears each day. Freshness of the head dominates user-visible quality: a stale price, a dead link in the top 3 results, a news story missing for 6 hours. The frontier is fed mostly by the recrawl scheduler, not by link discovery. Correctness bar: a per-tier freshness SLO.

Intent 2 — Bulk corpus crawl. Batch. Archives and ML training corpora want breadth, provenance, and cleanliness. Recrawl is rare; dedup and quality filtering dominate cost. Opt-out compliance is now a first-order concern — site owners increasingly publish rules specifically for AI crawlers, and a corpus that ignores them is a legal liability that outlives the crawl. Correctness bar: dup rate, opt-out compliance, and provenance per document.

Intent 3 — Targeted change monitoring. A fixed watch-list (5M product pages, 50K regulatory filings). No discovery. Latency-to-detect is the SLO, so fetch frequency per URL is high — which makes politeness harder, because you revisit the same hosts constantly. Noise suppression (ignore timestamps, ad slots, CSRF tokens) is the core algorithm. Correctness bar: time-to-detect p95 and false-change rate.

2.2 When NOT to Build a Crawler#

🎯 Staff Move: "Before we design this, I'd check whether a public crawl plus targeted top-up fetches covers the need. Building a general crawler is a multi-year commitment with a permanent on-call and an abuse-complaint inbox."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

Each fault line is a place where two competent engineers can disagree. The Staff-level signal is naming the options, the victim of each, and committing to a default with an explicit deviation trigger.

3.1 Fault Line 1: Politeness vs Throughput#

The tension: Every extra request per second to a host makes your index fresher and their server busier. You don't own their capacity, and they don't owe you tolerance.

Staff default: Adaptive delay with a floor of 1 s and one concurrent connection per registered domain, honoring 429/503 by doubling the delay (capped at 1 hour) and honoring Crawl-delay when present. An allowlist of the top ~1,000 hosts gets higher limits, each entry with a named owner and an expiry.

Per-IP, not just per-host: 10,000 small sites on one shared-hosting IP are one server. Keep a second politeness key on resolved IP (e.g., ≤ 4 concurrent connections per IP) — otherwise host-level politeness still melts the shared box.

When to deviate: Change-monitoring intents with contractual customer SLOs may justify a negotiated faster rate with the site owner — never a unilateral one.

🎯 Staff Move: "Politeness is a hard constraint, not a knob I trade for freshness. If the business needs a host crawled faster than it tolerates, that's a partnership conversation, not a config change."

3.2 Fault Line 2: Freshness vs Coverage#

The tension: With ~40M fetches a day, every recrawl of a known page is a fetch not spent discovering a new one — and vice versa.

Staff default: A fixed discovery floor (≥ 20% of daily fetches) so new content is never starved, and the rest allocated by importance × P(changed), with per-tier age SLOs as hard floors. Clamp the per-URL recrawl rate at the tier's minimum interval so hyper-volatile pages don't monopolize the budget.

The math worth saying out loud: With a Poisson change model, if a page changes at rate λ and we recrawl every interval I, its expected freshness is (1 − e^(−λI)) / (λI). A page changing daily (λ = 1/day) recrawled daily is fresh ~63% of the time; recrawled every 6 hours, ~88%. That diminishing return is why "crawl the head more" has a ceiling.

When to deviate: A bulk corpus crawl sets the recrawl share near 0%. A change-monitoring crawl sets discovery to 0%.

3.3 Fault Line 3: Centralized vs Host-Partitioned Frontier#

The tension: A single global priority order is easy to reason about; enforcing politeness across many workers is not.

Staff default: Host-partitioned, keyed by registered domain, with a per-IP secondary limit. Partitions (e.g., 4,096) mapped to workers by consistent hashing so rebalancing moves ~1/N of hosts. Frontier state persisted in a local embedded store (RocksDB) with periodic checkpoints to object storage, so a worker restart recovers in minutes rather than re-deriving from the URL-seen store.

When to deviate: Change monitoring on a small watch-list (< 10M URLs) can use a time-wheel scheduler per host without front queues.

3.4 Fault Line 4: Dedup Precision vs Cost#

The tension: Exact sets cost RAM and disk; probabilistic structures cost silent coverage loss.

Staff default: Exact URL-seen store with a Bloom filter as a negative cache in front — if Bloom says "never seen," it's definitely new; if "maybe seen," confirm against the exact store in a batched lookup. SHA-256 for exact content, SimHash for near-dups at the index boundary, with near-dup clusters keeping a canonical representative (respect rel=canonical when it agrees with our cluster).

🎯 Staff Move: "A Bloom filter's false-positive rate is a coverage-loss rate with no alarm attached. I'll use it to skip the disk lookup, never to make the final decision."

3.5 Fault Line 5: Crawl Everything vs Crawl Responsibly#

The tension: The corpus is more valuable the more it includes; the liability grows faster than the value at the edges.

Staff default: Follow RFC 9309 semantics: robots.txt 4xx → treat as "allow all"; 5xx or timeout → treat as "disallow all" temporarily, reuse the last good copy for up to ~24h if we have one, and retry with backoff. Opt-outs and takedowns flow through a policy service with a 24h propagation SLO, enforced at both fetch time and serving/export time — because data already in the corpus must also be purged.

Who signs off: Legal owns the policy; the crawl team owns enforcement and the propagation SLO; each downstream consumer (index, ML) owns purging its derived copies. Write that down before the first takedown arrives.

4. Failure Modes & Operational Reality#

4.1 Spider Trap — The Infinite Calendar#

t=0:       Host events.example.org added via a new link; budget default 50K/day
t=+10min:  Crawler finds /calendar?month=2026-10, each page links next month
t=+2h:     42,000 URLs discovered on this host; 99% are /calendar?month=...
t=+2h:     crawler.host_new_url_rate{host} = 350/min vs 30-day baseline 0
t=+3h:     Per-host budget exhausted; host drops to lowest band (damage capped)
t=+3h05m:  Trap detector flags pattern: 1 param varying, 97% near-dup ratio
t=+1day:   Pattern rule auto-generated: month > now+24 months → drop

• Detection: crawler.host_new_url_rate vs baseline, crawler.host_dup_ratio > 0.8, URL-pattern cardinality per path template.
• Blast radius: Without a per-host budget, one host can consume a partition's entire day — every other host on that partition goes stale. With budget: ~50K wasted fetches, one host.
• Mitigation: Per-host budget, pattern-based suppression, lowest-priority band for suspects.
• Prevention: Normalize known-infinite parameters; learn per-host URL templates; cap depth at 16.
• Owner: Crawl team on-call; trap-rule changes reviewed by the crawl quality owner.

4.2 IP Range Banned by a CDN#

t=0:       A large CDN's bot-management tightens thresholds
t=+15min:  403 rate for hosts behind that CDN rises 0.5% -> 22%
t=+15min:  Bodies are 200/403 challenge pages, ~5KB, identical SimHash
t=+30min:  crawler.http_status_ratio{status=403, asn=cdn} crosses 10% for 15m -> page
t=+45min:  On-call pauses the affected fetch pool; checks reverse-DNS verification
t=+4h:     Contact via CDN's verified-bot program; crawler IPs re-verified

• Detection: 403/429 ratio by destination ASN; soft-block detection (challenge-page fingerprint cluster).
• Blast radius: Every host behind that CDN — potentially 10–20% of the web — goes stale.
• Mitigation: Pause, don't hammer (retrying through a ban deepens it). Serve stale from index.
• Prevention: Publish crawler IP ranges and reverse-DNS; enroll in CDN verified-bot programs; separate IP pools per crawl purpose so an experimental crawl can't burn the search crawler's reputation.
• Owner: Crawl team for response; partnerships for CDN relationships.

4.3 DNS Resolver Saturation#

t=0:       New discovery wave: 2M never-seen hosts enter the frontier
t=+5min:   DNS cache hit rate drops 97% -> 60%
t=+8min:   Upstream resolver p99 3s; fetchers block on resolution
t=+10min:  crawler.fetch_rate down 70% although hosts are ready
t=+12min:  Alert on crawler.dns_latency_p99 > 500ms

• Detection: crawler.dns_cache_hit_ratio, crawler.dns_latency_p99, fetch rate vs ready-host count.
• Mitigation: Async DNS pre-resolution when a host enters the frontier (not at fetch time); rate-limit new-host admission; run local recursive resolvers per fetch cluster.
• Owner: Crawl infra.

4.4 Silent Staleness — The Scheduler Stops Scheduling#

The most dangerous failure: nothing errors. A deploy changes the importance-signal join key; 70% of URLs get importance 0; the scheduler dutifully recrawls the tail.

t=0:       Ranking team renames a field in the importance export
t=+1day:   Recrawl scheduler reads null importance -> defaults to 0 for 70% of URLs
t=+1day:   fetch_rate unchanged; error rate unchanged; dashboards green
t=+4days:  Top-tier p90 age: 20h -> 86h
t=+5days:  Search quality team reports stale results for news queries

• Detection: crawler.freshness_age_p90{tier} against SLO — the metric. Also a canary set: 10K known-volatile URLs whose last-fetch age is alerted directly. Schema checks on the importance feed (null-rate > 1% → reject the feed, keep last good).
• Blast radius: Whole index freshness, degrading slowly over days.
• Owner: Crawl team owns the SLO; ranking team owns the feed contract. The contract needs a schema and a consumer-driven test.

🎯 Staff Move: "Fetch rate is a vanity metric. The crawler's SLO is age of the head tier. If I could have one alert, it's freshness_age_p90{tier=top10M} > 24h for 2h."

4.5 Poison Content — The 2 GB "HTML" Page and the Parser Crash Loop#

• Symptom: Parser workers OOM repeatedly on one partition; the Kafka consumer lag for that partition grows.
• Root cause: A server streams an endless body, or a gzip bomb (1 MB → 10 GB).
• Detection: parser.oom_restarts, per-partition consumer lag, fetch.body_bytes_p99.
• Mitigation: Fetch-side cap (10 MB, truncate and mark), decompression ratio cap (e.g., 100×), parser in a sandbox with memory limit, poison records to a DLQ after 2 failures.
• Owner: Crawl team; DLQ reviewed weekly.

4.6 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Detailed HTML parsing knowledge ≠ crawler design. Knowing DOM quirks doesn't show you can allocate a fetch budget.
• Big storage estimates ≠ scale thinking. "100 TB" is arithmetic; "the top 1,000 hosts cap our head freshness at 86K fetches/host/day" is scale thinking.
• Naming Kafka for the frontier ≠ a frontier design. A log doesn't enforce per-host timing; the scheduler state is the design.
• Mentioning SimHash ≠ understanding near-dup. The question is what it costs when it's wrong (merged product variants).

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• HTML parsing details, character-set detection — "a library, sandboxed."
• TCP tuning and HTTP/2 multiplexing — mention keep-alive per host batch, move on.
• Ranking algorithms — importance is an input owned by another team.
• Exact storage arithmetic beyond one line.

6.4 Follow-Up Questions to Expect#

1. "How do you handle a site with 100M pages that allows only 1 req/s?" (≈ 3.2 years per full pass — prioritize, use sitemaps, negotiate.)
2. "How would you detect that a page is a soft 404?"
3. "What's the URL-seen store's lookup path at 10K new links/s?"
4. "How do you rebalance the frontier when a worker dies?"
5. "How would you crawl the same host from two regions without violating politeness?"
6. "How do you propagate a takedown to data already in the index and in ML datasets?"
7. "How would you measure coverage — what fraction of the web you're missing?"

7. Active Drills#

Drill 1: The Opening (Intent + Constraints)#

Prompt: "Design a web crawler."

Drill 2: The Throughput Ceiling#

Prompt: "You have 2,000 fetcher machines. How many pages per second can you crawl?"

Drill 3: Make the Frontier Concrete#

Prompt: "Walk me through exactly how a URL goes from discovered to fetched."

Drill 4: Dedup at 10 Billion URLs#

Prompt: "How do you know you've already seen a URL? And a page?"

Drill 5: The Spider Trap#

Prompt: "One site generates infinite URLs. What happens, and what do you do?"

Drill 6: Freshness Budget#

Prompt: "You can do 40M fetches a day across 1B known pages. How do you decide which?"

Drill 7: robots.txt Is Unreachable#

Prompt: "robots.txt for a major site times out. Do you crawl?"

Drill 8: Build vs Buy#

Prompt: "The ML team needs 5B clean English pages. Build a crawler?"

Drill 9: Changing the Politeness Policy Without an Incident#

Prompt: "Legal wants us to honor a new AI-crawler opt-out directive starting next week."

Drill 10: Multi-Region Crawl#

Prompt: "Latency to Asian hosts is 250 ms from us-east. Add a second crawl region?"

8. Deep Dive Scenarios#

Deep Dive 1: The Election-Night Freshness Incident#

Context: It's election night. Search results for "election results" show pages 3–6 hours stale. The top 200 news hosts update every 2–5 minutes. Search leadership escalates to you at 9 p.m.

Questions to Surface First:

• Is the crawler fetching these hosts and the indexer lagging, or is the crawler not fetching them?
• Are these hosts in the news tier with hourly SLOs, or were some mis-tiered?
• Are we being throttled (429/503) by the news sites themselves under their own traffic spike?
• What is the freshness SLO for this tier, and who agreed to it?

Typical L5 Approach: Checks the fetch rate (normal), adds fetchers, and bumps priority for the news domains. Fetch rate rises, but the news hosts are already at their politeness ceiling, so the extra capacity goes to the tail. Freshness doesn't move.

Staff Approach: Splits the pipeline: fetch age vs index age. Finds fetches are 10 minutes fresh but indexing of the news partition lags 4 hours — the indexer's queue is flooded by a scheduled full recrawl of a large forum host. Pauses the bulk recrawl, drains the news partition first, and adds a priority lane from crawler to indexer so news never queues behind bulk again.

Principal Approach: Asks why one freshness SLO depended on an unprioritized shared indexer queue, and why "election night" was a surprise. Institutes an event calendar (elections, sports finals, product launches) that pre-shifts recrawl budget and indexing capacity, and makes end-to-end freshness (live change → servable) the SLO owned jointly by crawl and index, with one dashboard and one error budget.

Key Takeaway: "Fetch rate says nothing about freshness. Measure age end-to-end, and split it at every queue."

What clears the Staff bar:

• Decomposes staleness into fetch vs index before acting
• Knows the news hosts are at the politeness ceiling, so more fetchers can't help
• Fixes the structural cause (shared lane) rather than the symptom

Deep Dive 2: The Silent Coverage Loss#

Context: A quarterly coverage audit samples 100K URLs from a third-party link dataset. 6% of them — on hosts we crawl regularly — are unknown to the crawler. Nobody noticed. You're asked why.

Questions to Surface First:

• Are the missing URLs unseen (never discovered) or seen-but-never-fetched?
• Is the loss uniform across hosts, or concentrated on some patterns (query strings, new sections)?
• When did the URL-seen implementation or the normalizer last change?

Typical L5 Approach: Adds the missing URLs as seeds and increases discovery budget.

Staff Approach: Classifies the missing URLs. 70% were "seen" according to the URL-seen path but have no fetch record — a normalizer change 4 months ago started stripping a query parameter (?page=) that some hosts use for real content, collapsing distinct pages into one canonical URL. The remaining 30% are Bloom false positives from a filter that grew to 3× its design capacity, pushing FPR from 1% to ~15% because nobody resized it. Fixes: revert the normalization rule to a per-host allowlist, rebuild the filter at the right size, and add the exact-store confirmation path the design should always have had.

Principal Approach: Treats coverage as an unmeasured SLO — the real failure. Funds a permanent coverage-sampling pipeline (weekly, against external link datasets and sitemaps), makes normalizer changes a reviewed, versioned, shadow-tested change class, and adds capacity alarms for every probabilistic structure (bloom.fill_ratio > 0.5 → ticket).

Key Takeaway: "Every probabilistic or lossy step in a crawler is a place pages disappear silently. Each needs a sampled ground truth."

What clears the Staff bar:

• Classifies before fixing
• Treats normalization as a data-loss risk, not a cleanup
• Adds a ground-truth measurement, not just a fix

Deep Dive 3: Onboarding a Mega-Host#

Context: The company signs a partnership with a marketplace with 400M product pages. They want everything indexed within 30 days and fresh prices daily. Their current allowance is 1 req/s.

Questions to Surface First:

• 400M pages at 1 req/s = ~12.7 years. What rate will they actually sustain?
• Do they publish sitemaps with lastmod, or a product feed?
• Which pages matter — all 400M, or the ~20M with inventory?

Typical L5 Approach: Raises this host's rate limit to 200 req/s so 400M pages crawl in ~23 days.

Staff Approach: Asks for a feed. A daily product-change feed (or sitemaps with accurate lastmod) turns "crawl 400M pages daily" into "fetch the ~5–10M that changed." Negotiates a crawl rate with their SRE team (e.g., 50 req/s off-peak, 10 req/s peak, scheduled windows), registers it in the allowlist with an owner and expiry, and ramps: 10 → 25 → 50 req/s over a week watching their latency. Initial backfill targets in-stock pages first.

Principal Approach: Recognizes this as the first of many partner integrations and designs a "structured ingest" path beside the crawler — feeds and APIs with partner SLAs — so partners stop being special-cased allowlist entries. Prices the choice: feed ingest costs ~1/50th the fetches of a crawl for the same freshness.

Key Takeaway: "The fastest crawl of a large site is the one you replace with a feed."

What clears the Staff bar:

• Does the 12.7-year math before proposing anything
• Reaches for feeds/sitemaps over brute force
• Ramps rate with gates and an expiry

Deep Dive 4: Post-Mortem — We Took Down a Hospital's Website#

Context: A regional hospital's site went down for 40 minutes. Their logs show 60% of requests came from our crawler. It was on shared hosting with 3,000 other sites on one IP. This hits the press. You own the post-mortem.

Questions to Surface First:

• What was our request rate to that IP (not host) during the window?
• Did per-host politeness hold? (Probably yes — for each of the 3,000 hosts.)
• Did we honor 503/429 and response-time backoff?

Typical L5 Approach: Adds the hospital's domain to a blocklist and lowers its rate.

Staff Approach: Identifies the root cause: politeness was keyed only on host. A discovery wave admitted 400 of the 3,000 co-hosted sites at once; each was individually polite (1 req/s), so the shared IP took ~400 req/s. Fix: add a per-IP concurrency and rate limit (e.g., ≤ 4 concurrent, ≤ 5 req/s per IP), resolved at admission time; make response-time backoff apply at the IP level; add an alert on per-IP request rate.

Principal Approach: Treats it as a policy failure: the politeness standard never defined the unit of "a server." Updates the org's crawl standard to require host and IP limits for every crawler in the company, adds a public complaint channel with a 1-hour response SLA, and runs a quarterly game day simulating shared-hosting load.

t=0:        Discovery wave admits 400 hosts on one shared IP
t=+2min:    Per-IP rate 400 req/s; server p99 800ms -> 9s
t=+6min:    Server returns 503; host-level backoff kicks in per host, slowly
t=+10min:   Site down; our per-host dashboards show nothing abnormal
t=+40min:   Hospital's host provider blocks our range; site recovers
t=+2h:      Complaint reaches us via cloud provider abuse desk

Key Takeaway: "Politeness has to be keyed on the thing that falls over — the server — not the name you used to reach it."

What clears the Staff bar:

• Finds the keying flaw, not a one-off blocklist
• Separates response, fix, and standard change
• Treats the ecosystem relationship as part of the incident

Deep Dive 5: Multi-Region Expansion and Data Residency#

Context: The company launches search in the EU and Asia. Local results quality is poor: regional sites are crawled from us-east with 200–300 ms RTTs, some geo-block US IPs, and legal flags that EU-origin content must be stored in-region.

Questions to Surface First:

• What fraction of target hosts geo-block or serve different content to US IPs?
• Does residency apply to raw fetched bytes, derived index, or both?
• Does global dedup require cross-region content comparison?

Typical L5 Approach: Deploys a complete crawler copy in each region with its own URL sets.

Staff Approach: Assigns each host a home region (geo-IP + measured RTT + geo-block detection). Frontier partitions and WARC buckets live in the home region, so politeness stays single-writer and bytes stay in-region. Global state is small and metadata-only: URL-seen hashes, SimHash fingerprints, host assignments. Cross-region dedup compares fingerprints, never content.

Principal Approach: Decides residency is a platform capability, not a crawler feature: a data-classification tag on every stored object, enforced by storage policy for every downstream consumer. Weighs the one-way door: once derived datasets mix regions, separating them later means rebuilding from raw — so the tag must exist from day one.

Key Takeaway: "Regionalize bytes and politeness; globalize only fingerprints and policy."

What clears the Staff bar:

• Keeps single-writer politeness across regions
• Moves metadata, not content, across borders
• Treats residency as a storage tag enforced downstream

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Name the three crawler intents and explain why each needs a different scheduler
• Derive the throughput ceiling from host count and politeness delay, not fetcher count
• Draw a host-partitioned frontier with front/back queues and explain why politeness needs no lock
• Separate URL-seen, exact-content, and near-dup deduplication, and name the victim of each false positive
• Allocate a daily fetch budget with a discovery floor and importance × P(changed)
• Defend against spider traps with budgets first and heuristics second
• State RFC 9309 robots.txt failure semantics and who owns the policy
• Name the one freshness metric you'd page on
• Price the crawler at three scales and name its one-way doors

The Bar for This Question#

Mid-level (L4): Builds the fetch-parse-enqueue loop, handles retries, stores pages. Mentions robots.txt when prompted. Treats dedup as a hash set.

Senior (L5): Adds a distributed priority queue, per-domain rate limiting, a Bloom filter, and a recrawl interval. Sizes storage correctly. The design works but spends budget uniformly, keys politeness wrongly for shared hosting, and has no freshness SLO — it fails quietly.

Staff+ (L6): Commits to an intent, frames the design around a fetch budget, makes politeness structural via host partitioning, layers dedup with explicit silent-failure owners, defends traps with per-host budgets, and measures freshness end-to-end with named metrics and owners. Talks about site operators as stakeholders. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "The Crawler's Throughput Number Is Meaningless"#

The Staff position: Report freshness age per tier and coverage ratio. Throughput is a capacity metric for the infra team, not a success metric.

Why this matters in interviews: Candidates who optimize pages/s signal they haven't owned a crawler's outcome.

10.2 "Bloom Filters Don't Belong on the Decision Path"#

The Staff position: Bloom as a negative cache in front of an exact store. Never the authority.

Why this matters in interviews: It shows you reason about failure visibility, not just memory efficiency.

10.3 "Most Companies Should Never Build a General Crawler"#

The Staff position: Build the processing pipeline and targeted fetchers; buy or reuse coverage.

Why this matters in interviews: When asked "build vs buy," the strong answer questions the premise.

10.4 "Politeness Is a Product Feature"#

The Staff position: Budget engineering time for crawler identity, complaint handling, and per-IP limits as first-class features.

Why this matters in interviews: It reframes an "ethics footnote" as an availability requirement.

10.5 "Raw Bytes Are the Only Durable Asset"#

The Staff position: Write WARC first, parse downstream, keep raw for a defined retention owned by legal.

Why this matters in interviews: It turns "what if the parser has a bug?" into a one-sentence answer.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer designs a crawler. A Principal engineer notices that search, ML data, trust & safety, ads quality, and a price-intelligence team each run their own crawler — five bot identities, five politeness implementations, five legal postures, all hitting the same hosts from the same cloud IP ranges. The real problem is that the organization's relationship with the open web is fragmented: one team's aggressive experiment gets the whole company's IP range banned, and a takedown request has to be honored by five systems nobody inventoried. The L7 move is to decide what is centralized (identity, politeness, policy, raw storage) and what stays per-team (schedulers, parsers).

The Org-Level Fault Line#

One crawl platform vs per-team crawlers.

The L7 default: Centralize the fetch layer, politeness state, bot identity, policy service, and raw WARC store. Consumers submit fetch requests with priorities under a per-consumer quota; each keeps its own scheduler. Chargeback per million fetches makes the cost visible.

🧭 Principal Move: "I'd centralize everything that touches someone else's server — identity, politeness, policy — and decentralize everything that expresses a team's product judgment, like scheduling and parsing."

Cost Model#

Assumptions: cloud list prices, ~100 KB average raw page, ~25 KB compressed, 3-month raw retention, fully loaded engineer cost ~$25K/month, inbound bandwidth free, public-IP fetchers (no managed NAT per-GB charges).

The number that surprises executives: at small scale, people cost 3–4× the infrastructure. At large scale, JavaScript rendering alone can exceed the rest of the crawler combined. The most expensive decision at every scale is "render everything."

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC: Web Crawling Standard v1

Scope: Any system in the company that issues automated HTTP requests to hosts we don't own.

MUST:

• Use the shared fetch platform or a registered bot identity with a documented User-Agent, contact URL, and reverse-DNS-verifiable IPs.
• Enforce politeness per registered domain and per resolved IP: ≤ 1 concurrent connection per domain, ≤ 4 per IP, ≥ 1 s spacing by default.
• Honor robots.txt per RFC 9309 (5xx/timeout = disallow; cached copy ≤ 24 h), 429/503 Retry-After, and registered opt-out directives.
• Propagate takedowns to all stored and derived data within 24 h and report completion to the policy service.
• Store provenance (URL, fetch time, robots state) with every stored document.

SHOULD:

• Prefer feeds, sitemaps, and APIs over crawling when available.
• Use conditional GET for all recrawls.

Exceptions: Filed with the crawl platform team; approved by crawl platform lead + legal; expire after 90 days.

Success metrics: Abuse complaints per billion fetches < 1; takedown propagation p99 < 24 h; zero unregistered crawlers found in quarterly egress audits.

What I'd Tell the VP#

We crawl the web for five different products, and today each team does it separately — which means five ways to get our IP addresses banned and five places a legal takedown can be missed. I'm proposing one shared crawl platform that owns our identity, our manners toward other sites, and our compliance, while each product keeps control of what it wants to crawl. It costs roughly two additional engineers for a year and reduces duplicate fetching by about a third. The biggest risk we remove is a single team's mistake taking every product's data supply offline. The biggest cost we avoid is rendering JavaScript for everything; we'll render only the ~5% of pages where it changes results.

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll build a great crawler for search" — correct, but ignores the four other teams crawling the same hosts.
• "Legal owns robots policy" — names an owner but doesn't design the propagation path or audit.
• "Freshness SLO p90 < 24 h" — a good SLO with no price attached; L7 says what it costs and what the next tier would cost.

Appendices

Appendix A: Mechanics in Depth#

A.1 URL Normalization#

normalize(url):
  u = parse(url)
  u.scheme = lower(u.scheme); u.host = lower(idna_encode(u.host))
  drop default port (80 for http, 443 for https)
  u.path = remove_dot_segments(u.path); percent-decode unreserved chars
  drop fragment
  params = [p for p in u.query if p.name not in SESSION_PARAMS
                                and not p.name.startswith("utm_")]
  params = sort(params) unless host in ORDER_SENSITIVE_HOSTS
  return serialize(u)

Why it's dangerous: every rule is a potential collapse of distinct pages. Rules are versioned; per-host exceptions exist; changes are shadow-tested with a collapse report (Deep Dive 2).

A.2 Adaptive Politeness#

on_fetch_complete(host, response_time, status):
  if status in (429, 503):
      host.delay = min(host.delay * 2, 3600s)
      if retry_after present: host.delay = max(host.delay, retry_after)
  else:
      target = max(host.min_delay, 10 * response_time)
      host.delay = 0.8 * host.delay + 0.2 * target   # smoothed
  host.next_fetch_at = now + host.delay
  ip.next_fetch_at   = max(ip.next_fetch_at, now + ip.min_delay)

A.3 URL Lifecycle#

A.4 Change-Rate Estimation#

A simple, robust estimator: over the last n fetches at intervals Iᵢ, with X changes observed, λ ≈ −ln(1 − X/n) / mean(I) (valid when X < n). Seed new URLs with a prior from their host and path template. Clamp λ to the tier's min and max recrawl interval.

A.5 SimHash#

simhash(tokens):
  v = [0] * 64
  for t, weight in shingle_weights(tokens):
      h = hash64(t)
      for i in 0..63: v[i] += weight if bit(h, i) else -weight
  return bits(v[i] > 0 for i in 0..63)
near_dup(a, b) = popcount(a XOR b) <= 3

Index: split 64 bits into 4 blocks of 16; store 4 tables each keyed by one block. Any pair within distance 3 matches exactly in at least one block (pigeonhole), so lookups are 4 exact probes plus candidate verification.

Appendix B: Keys and Data Model#

Appendix C: Coordination Mechanisms#

Quick comparison: a distributed lock per host (Redis) works to ~10K fetches/s and becomes the hot path; partition ownership scales linearly with no per-fetch coordination. Use locks only for rare administrative operations. See Distributed Coordination.

Appendix D: Client (Crawler) Behavior Contract#

Appendix E: Observability#

E.1 Core Metrics#

Outcome:     crawler.freshness_age_p90{tier}, crawler.coverage_sample_hit_ratio
Budget:      crawler.fetches_by_reason{discovery|recrawl|feed}, crawler.budget_denied{tier}
Politeness:  crawler.host_delay_p50, crawler.ip_request_rate_max, crawler.http_status_ratio{status}
Health:      crawler.dns_cache_hit_ratio, crawler.fetch_latency_p99, frontier.disk_used_pct
Quality:     crawler.dup_ratio{exact|near}, crawler.trap_suspects_open, crawler.soft404_ratio
Ecosystem:   abuse.complaints_open, policy.takedown_propagation_age_p99

E.2 Critical Alerts#

E.3 Control Plane vs Data Plane#

Control plane: policy service, recrawl scheduler, host assignment, allowlists — changes rolled out shadow → canary → enforce. Data plane: frontier partitions, fetchers, parsers — must keep running with last-known-good policy if the control plane is down (fail-static), except that robots/takedown enforcement never fails open.

E.4 Debugging Silent Staleness#

1. Split age by stage: live change → fetched → parsed → indexed.
2. Check the scheduler's inputs: importance-feed null rate, λ distribution shift.
3. Check budget denial by tier — is the top tier being denied?
4. Check canary URLs (10K known-volatile pages) directly.

Appendix F: Scale Evolution#

F.1 What Works at Each Scale#

F.2 Multi-Region Path#

Start with one region. Add a second when > 20% of target hosts are > 150 ms away or geo-block. Assign hosts home regions; forward links; keep global only hashes, fingerprints, and policy (Drill 10, Deep Dive 5).

F.3 What You Don't Build on Day One#

• JavaScript rendering (target 1–5% later)
• Multi-region fetch
• Learned change-rate models (start with the Poisson estimator)
• Near-dup across all history (start with within-host)
• A general platform API (until a second consumer exists)

Appendix G: Multi-Tenancy, Fairness, and Cost#

When the fetch layer serves multiple consumers:

🧭 Principal Insight: "The site operator should never be able to tell how many teams we have. One identity, one budget, one set of manners."

Related reading: Distributed Rate Limiting for per-key budgets, Message Queues for the fetch-result pipeline, Search Indexing for the downstream consumer, Task Scheduling for time-based scheduling, Consistent Hashing, and Data Pipelines.