Technologies referenced in this case study: Elasticsearch · Apache Kafka · PostgreSQL · Redis · Flink & Stream Processing

Related: Database Indexing · Sharding & Partitioning · Scaling Reads · Data Pipeline Patterns · Web Crawler · Database Sharding

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once. Return to individual sections for targeted review.

With SQL LIKE on the product table:
t=0:       Catalog at 40M products; query "red running shoes"
t=+0s:     WHERE title ILIKE '%red%running%shoes%' → full table scan, 9 s
t=+1wk:    Search traffic 800 QPS; primary DB CPU 100%; checkout slows
t=+2wk:    "running shoe red" returns nothing — word order and stemming unsupported
t=+1mo:    Conversion from search down 18%; a read replica dedicated to LIKE queries

With an inverted index (20 shards × 3 replicas, NRT refresh 1 s):
t=0:       Same query → 3 posting lists intersected, BM25 top-1000, re-ranked to top-50
t=+0s:     p50 25 ms, p99 120 ms at 3,000 QPS; primary DB untouched
t=+1s:     Price change on a product visible in search within ~2 s via CDC
t=+1mo:    Learned ranking lifts search conversion 6%; zero impact on checkout DB

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What This Interview Actually Tests#

Search is not an inverted-index question. Everyone can describe posting lists.

It is a fan-out, freshness and relevance-ownership question that tests:

• Whether you shard for the query path (scatter-gather tail latency) and not just for data size
• Whether you state a freshness SLO and pay for it deliberately (refresh interval, indexing pipeline)
• Whether you can reindex a live index with a schema or analyzer change without downtime
• Whether you treat ranking as a product surface with offline metrics, online experiments and owners
• Whether the index is a derived store you can rebuild — and who owns the rebuild

The key insight: The index is a cache of the source of truth, optimized for a query shape. Staff engineers design it to be rebuilt, swapped and ranked independently — and they own the gap between what's in the database and what's findable.

The L5 vs L6 Contrast — Start Here#

The Staff Positions#

The Three Intents#

🎯 Staff Move: "I'll design e-commerce product search, because it forces freshness, ranking and reindexing into the same conversation. Web-scale search changes the index build to batch tiers; enterprise search makes permissions the hardest problem — I'll point out where those diverge."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Citing specific production systems demonstrates you've studied operational reality, not textbook designs.

Google — Tiered Indexes and Incremental Indexing (Percolator / Caffeine)#

Google's Percolator paper (2010) describes moving web indexing from large MapReduce batch rebuilds to incremental processing with cross-row transactions over Bigtable, powering the Caffeine indexing system and cutting the average age of documents in search results substantially. The broader pattern — a fast-updating fresh tier alongside a larger, slower-built base tier — is how web-scale engines balance freshness with build cost.

Staff insight: Freshness and index build cost are a tier decision, not a single knob. Name "fresh tier + base tier" and you've shown you know web-scale search isn't one index.

LinkedIn — Galene#

LinkedIn publicly described Galene, its Lucene-based search architecture, which builds base indexes offline in Hadoop and serves them alongside a live-update index for recent changes, with a federated broker fanning out to verticals (people, jobs, companies) and multi-stage ranking.

Staff insight: Offline-built base index + live update layer + federation is the production answer to "how do you reindex 1B documents without downtime?" — you don't reindex live; you build and ship snapshots.

Twitter — Earlybird#

Twitter's Earlybird (2012 paper) is a real-time inverted index optimized for tweets searchable within seconds: in-memory, append-only posting lists partitioned by time into segments, with concurrency designed for a single writer and many readers.

Staff insight: When freshness is the product, the index layout changes — time-partitioned, append-only, recency-ordered posting lists. Designing for the dominant query ("recent tweets matching X") beats a general-purpose index.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: Writes flow from sources of truth through CDC and enrichment into a bulk indexer; the search cluster never receives writes directly from product code. Every document carries a version (source LSN or updated_at) so out-of-order updates can't regress it. Reads go through an alias, fan out to one copy of each of 20 shards, merge top-K, and re-rank a small candidate set. Signals (popularity, CTR) arrive via a slower batch path — they don't need second-level freshness.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 15 minutes explaining tokenization, TF-IDF and posting-list compression, then have no time for freshness, reindexing or ranking evaluation. The inverted index is table stakes — get past it in 3 minutes.

Phase 1: Requirements & Framing (2–3 min)#

State scope in one sentence:

"Users type a query and get the top 50 relevant products with filters and facets; sellers update products and prices continuously."

Non-functional constraints that pick the design:

"The numbers that decide the design: corpus size and growth, update rate and freshness SLO per field, query QPS and latency, and how relevance is measured. I'll assume 50M products (~2 KB each searchable, ~100 GB raw), 2K updates/s with bursts to 20K during repricing, price/stock visible within 5 s, 5K QPS peak, p99 ≤ 200 ms, and relevance measured by search conversion."

Commit:

"I'll design product search. Freshness matters for price and stock only; popularity signals can be a day old."

🎯 Staff Move: Ask how relevance is judged in Phase 1. It tells the interviewer you know search quality is measured, not asserted — and it sets up the ranking deep dive.

Phase 2: Core Entities & API (1–2 min)#

• SearchDocument (product_id, version, title, description, brand, category_path, price, in_stock, seller_id, popularity, embedding?)
• Index version (products_v7) behind alias products
• Query (text, filters, sort, page_token, user_context)
• Judgment (query, product_id, grade 0–3) for offline evaluation

Search(q, filters, sort, page_token, ctx) → { results[50], facets, total_estimate, next_page_token }
Suggest(prefix) → [completions]                 # separate, prefix-optimized index
IndexDocument(doc, version)                     # internal: pipeline only, idempotent by (id, version)

"Writes are internal and versioned — no product service writes to the search cluster directly."

Phase 3: High-Level Architecture (≤5 min)#

Walk the flow:

1. Product changes are captured by CDC into Kafka; enrichment joins seller/category/stock into a denormalized search document.
2. The bulk indexer writes to the alias's current index with external versioning (older versions rejected).
3. Queries hit the API: parse, spell-correct, expand synonyms, check the result cache.
4. Coordinator fans out to one copy of each shard; each returns its top-K by BM25 + features; coordinator merges.
5. L2 re-ranker scores the top 200 with a learned model; business rules (sponsored slots, diversity) applied last.

🎯 Staff Move: "This is the standard design. What decides quality and reliability is how fresh the index is and what that costs, how we change the index without downtime, and how we know ranking changes are improvements. Let me go there."

Phase 4: Transition to Depth (1 min)#

"Three deep dives: the freshness pipeline and its cost, sharding and tail latency under fan-out, and reindexing plus ranking changes as safe rollouts. I'd start with freshness — it's where stale prices turn into customer complaints. Your preference?"

Phase 5: Deep Dives (25–30 min)#

Deep dive 1: Freshness pipeline (7–8 min)

"End-to-end freshness = CDC lag (~200 ms) + enrichment (~100 ms) + bulk batching (up to 1 s) + refresh interval (1 s) ≈ 2–3 s p50, 5 s p99. The costly part is refresh: each refresh makes a new segment per shard; with 20 shards that's 20 segments/s being created and merged. At 2K updates/s it's fine; at 20K updates/s during a repricing burst, merge pressure competes with queries. So price updates use partial updates, and during declared bulk operations we raise refresh to 5 s for affected indexes."

Versioning: "Each doc carries the source LSN as an external version. If the CDC consumer replays or reorders, older versions are rejected — no stale overwrite."

Deep dive 2: Sharding and fan-out (6–8 min) — 100 GB raw → ~150 GB index → 20 shards of ~7.5 GB now, sized to ~30 GB at 4× growth. Replicas for QPS. Adaptive replica selection, timeouts with partial results, hedged requests for the slowest shard.

Deep dive 3: Reindexing with alias swap (6–8 min) — v8 build, dual-write, parity test, swap, 48-h rollback window.

Deep dive 4: Ranking evaluation (5–7 min) — judged query set, NDCG@10 offline gate, interleaving then A/B with guardrails (latency, zero-result rate, revenue per search).

Phase 6: Wrap-Up (2–3 min)#

"Summary: CDC-fed, versioned documents; 20 shards × 3 copies behind an alias; 1 s refresh with per-field freshness SLOs; multi-stage ranking with offline and online evaluation; versioned reindex with alias swap. Next I'd add hybrid retrieval with embeddings for long-tail queries and query understanding (category prediction). I wouldn't build a custom engine or a term-partitioned index at this scale."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Search sits between many owners: source-of-truth teams who change schemas, a platform team running clusters, product teams who want their items ranked higher, and a relevance team trying to optimize one metric without breaking five others. The technical core is well-known; the Staff skill is making the index rebuildable, freshness measurable, and ranking changes governed — and keeping p99 flat while fan-out grows.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"If we had to rebuild the entire index from scratch tomorrow, how long would it take, and would anyone notice?"

If the answer is "days, and yes," the index has become a source of truth by accident — documents written only to search, no versioning, no alias. Every Staff design choice (CDC from the source, versioned documents, aliases, parity tests) exists to make the answer "a few hours, and no."

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

E-commerce / product search. Corpus 10M–1B items; updates dominated by price/stock; ranking optimized for conversion with many business constraints (sponsored, diversity, availability). Freshness of transactional fields matters more than of text.

Web-scale / content search. Billions of documents; crawl-driven ingestion; index build as a batch pipeline producing immutable snapshots shipped to serving; a fresh tier for news/recent content. Serving is massively replicated; the challenge is fan-out at thousands of shards and index build cost.

Multi-tenant enterprise search. Many tenants with power-law sizes; per-document ACLs; strict isolation. Query must filter by tenant and permissions with zero leakage; small tenants share indexes with routing, large tenants get dedicated ones. The correctness bar is security, not relevance.

🎯 Staff Move: "The permission model is the design driver for enterprise search — a single missed ACL filter is a data breach. For product search, it's freshness of price and stock. I'll name which one I'm optimizing."

2.2 When NOT to Use a Search Engine#

"If a user edits a listing and immediately views 'my listings,' I serve that page from the database, not search — refresh lag would look like data loss."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: Freshness vs Indexing Cost#

The tension: Each refresh creates segments that must be merged; each update is a delete + reinsert in Lucene-family engines. Faster freshness means more CPU and I/O on the same nodes serving queries.

Staff default: Per-field freshness SLOs. CDC + 1 s refresh for fields used in filters/sort (price, in_stock). Display-only volatile data fetched at query time for the final 50 results. Signals batch-updated nightly via partial updates. Raise refresh interval during declared bulk operations.

When to deviate: Web-scale: batch-built base index + small fresh tier. Social/real-time: time-partitioned append-only index (Earlybird-style).

🎯 Staff Move: "I don't want 'real-time search' — I want price and stock within 5 seconds and everything else within a day. That's a third of the indexing cost."

3.2 Fault Line 2: Sharding — Fan-Out vs Shard Size#

The tension: Document-partitioned search sends every query to every shard. More shards → smaller shards (faster recovery, more parallelism per query) but more fan-out (more tail risk, more coordinator merge work, more per-shard overhead).

Tail latency controls:

• Adaptive replica selection: route each shard request to the copy with the lowest recent latency/queue.
• Per-shard timeout with partial results: return with 19 of 20 shards if one exceeds 150 ms; flag partial=true, log it.
• Hedged requests: after p95 latency, send a duplicate to another replica; take the first.
• Result cache for head queries (60 s TTL).

Staff default: Shards sized 10–50 GB at the 18-month projected size; replicas to meet QPS with 2× headroom; replica selection + hedging + partial-result timeouts. Shard count set once per index version (changing it = reindex, or split/shrink operations).

3.3 Fault Line 3: Relevance vs Latency#

The tension: Better models (cross-encoders, many features, personalization) improve ranking but cost milliseconds and CPU/GPU per candidate.

Staff default: Multi-stage. Put the expensive model only on the top 200. Under load, skip L2 (serve L1 order) before timing out: a slightly worse ranking beats an error page.

Degradation ladder: full pipeline → skip personalization features → skip L2 → reduce per-shard depth to 200 → serve cached results for head queries → reject tail queries with 503. Each step is a flag with a named owner.

3.4 Fault Line 4: Reindexing — In-Place vs Versioned Rebuild#

The tension: Mapping and analyzer changes can't be applied in place. Even when possible, updating 50M documents in the live index competes with queries and can't be rolled back.

Staff default: Always versioned indexes behind aliases; reindex from the source of truth (not from the old index if the old one might be wrong); dual-write via the CDC consumer; parity testing with a judged/top-query set; 48-h rollback window.

Parity test: run the top 10K queries against v7 and v8; compare result overlap (Jaccard@10) and NDCG@10 on judged queries; expected change (for an analyzer change) is declared up front — e.g., "overlap ≥ 0.8 except queries with plurals."

3.5 Fault Line 5: Platform vs Per-Team Search#

Staff default: Search platform owns clusters, indexing pipeline, alias/versioning tooling and evaluation infrastructure; product surfaces own their ranking models and business rules within latency budgets; source teams own document contracts.

🧭 Principal Move: "Relevance is a product with a P&L impact. I'd give each surface a ranking owner and a shared experiment platform — and a rule that no ranking change ships without an experiment readout."

4. Failure Modes & Operational Reality#

4.1 Merge Storm During Bulk Repricing#

Scenario: A seller-tools feature lets merchants reprice entire catalogs. A promotion triggers 30M price updates in 40 minutes.

t=0:       Repricing starts; update rate 2K/s → 12K/s
t=+2min:   Refresh at 1 s creates small segments on all 20 shards; merge threads saturate disk
t=+5min:   Query p99 120 ms → 900 ms; coordinator timeouts cause partial results on 8% of queries
t=+8min:   Search thread pool queues fill; shard.rejected_requests > 0
t=+10min:  Conversion from search drops 11%
t=+15min:  On-call throttles the indexer; freshness lag grows to 20 min for all products
t=+50min:  Backlog drains; p99 recovers

Detection: search.p99_ms > 300; index.merge_time_ms spike; shard.rejected_requests > 0; index.freshness_lag_s growth. Blast radius: All search users; all product freshness during throttling. Mitigation: Priority lanes in the indexer: price/stock partial updates first, bulk repricing lane rate-limited; temporarily raise refresh to 5–10 s. Prevention: Bulk operations are declared events with an ingestion budget; separate indexing and query node roles (or dedicated ingest capacity) at larger scale. Owner: Search platform (indexer lanes); seller-tools team (declares bulk jobs).

🎯 Staff Move: "Freshness and query latency share the same disks. I protect queries first and make bulk writers queue — and I make them tell us before they start."

4.2 Silent Staleness — The Stuck CDC Consumer#

t=0:       Enrichment consumer hits a poison message (malformed seller record); retries forever
t=+0s:     Partition 7 of products topic stops advancing; 1/32 of products stop updating
t=+3h:     Customer sees "$49" in search, "$79" at checkout — 12K complaints over the day
t=+9h:     Support escalates; consumer lag on partition 7 = 3.1M messages

Why it's silent: Search is up, latency is fine, most products update. Only a per-partition lag metric or an end-to-end freshness probe reveals it. Detection: consumer_lag{partition} max > 60 s; synthetic freshness probe (update a canary product every minute, measure time-to-searchable) — index.freshness_lag_s p99 > 30 s pages. Mitigation: Poison messages to DLQ after 3 retries; per-partition lag alerts. Prevention: Canary documents per partition; checkout re-validates price from source of truth (search is never authoritative for price). Owner: Search platform (pipeline); catalog team (bad records).

4.3 Reindex Swap Regression#

Scenario: v8 added a new analyzer with aggressive stemming. Alias swapped after doc-count verification only.

t=0:       Alias swap v7 → v8
t=+10min:  "glasses" now matches "glass" — drinking glasses outrank eyeglasses
t=+1h:     Zero-result rate unchanged, but search conversion −4.5% on eyewear
t=+6h:     Category manager notices; rollback via alias swap in 30 s

Lesson: Counts prove completeness, not quality. Parity tests on top queries + judged NDCG + a small online A/B before full swap. Detection: search.conversion_rate by category vs baseline; search.ctr_at_10 drop; parity jaccard@10 distribution. Owner: Relevance team (quality gate); platform (swap tooling).

4.4 Hot Shard from Custom Routing#

A marketplace routes by seller_id to make "search within store" single-shard. One mega-seller holds 18% of listings → one shard at 3× others' size and query load. Detection: shard.size_gb max/median > 2; shard.query_latency_ms skew. Mitigation: Routing partitions (route a key to a set of N shards), or dedicated index for mega-sellers. Owner: Search platform.

4.5 ACL Filter Missing (Enterprise)#

A new query path for "recent documents" is added without the permission filter. Users see titles of documents they can't open. Detection: Automated permission tests per query path (a canary user must never see canary restricted docs); search.acl_filter_missing_total from a query-builder assertion. Mitigation: All queries built by one library that injects tenant + ACL filters; direct engine queries prohibited. Owner: Search platform + security.

4.6 Coordinator Memory Blow-Up — Deep Pagination#

A scraper requests from=50000&size=100. Each shard returns 50,100 candidates; the coordinator sorts 1M entries per request; heap pressure triggers GC pauses cluster-wide. Mitigation: Cap from + size (e.g., 10K); use search-after cursors; rate-limit anonymous deep pagination. Owner: Search API team.

4.7 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Deep Lucene internals (FST, skip lists) ≠ search system design.
• Vector search enthusiasm — "use embeddings" without recall/latency/cost tradeoffs or a hybrid plan.
• Elaborate query parsing while ignoring the indexing pipeline.
• Naming Learning-to-Rank without feature freshness, training data bias or evaluation.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

6.4 Follow-Up Questions to Expect#

1. "How do you guarantee an updated price shows within 5 seconds, and how do you know it does?"
2. "Walk me through changing the mapping on a live index."
3. "How many shards and replicas, and why?"
4. "How do you handle a query that times out on one shard?"
5. "How do you evaluate a new ranking model before launching it?"
6. "How would you add semantic/vector search?"
7. "How do you handle deletions and GDPR erasure in the index?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design search for our e-commerce site."

Drill 2: Inverted Index Mechanics That Matter#

Prompt: "How does an update to a product become searchable?"

Drill 3: Make Freshness Concrete#

Prompt: "Prove to me a price change shows up within 5 seconds."

Drill 4: The Cluster Is Degraded#

Prompt: "Two of 12 data nodes die. What happens?"

Drill 5: Hot Tenant / Hot Shard#

Prompt: "Marketplace search 'within store' is slow for our biggest seller."

Drill 6: Multi-Tenant Enterprise Search#

Prompt: "Build search for a SaaS document product with per-document permissions."

Drill 7: Build vs Buy#

Prompt: "Self-host OpenSearch, use a managed service, or a SaaS search API?"

Drill 8: Change Ranking Without an Outage#

Prompt: "The team has a new ranking model. Ship it."

Drill 9: Cost#

Prompt: "The search cluster costs $220K/month. Cut it."

Drill 10: Multi-Region#

Prompt: "Serve search from 3 regions."

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Latency Collapse#

Context: Cyber Monday. Search QPS is 3.2× normal. p99 went from 140 ms to 1.4 s; 6% of queries return errors. On-call escalates to you.

Questions to Surface First:

• Is the bottleneck query CPU, coordinator merge, merges/indexing, or the re-ranker?
• Is traffic organic or a scraper doing deep pagination?
• What's the result-cache hit rate?
• Is indexing volume also elevated (promo repricing)?

Typical L5 Approach: Add nodes. New nodes need shard relocation, which adds recovery I/O during the peak — making things worse for 30+ minutes.

Staff Approach: Apply the degradation ladder immediately: skip L2 re-ranking (saves ~25 ms and a service hop), raise head-query cache TTL from 60 s to 5 min, cap from+size at 1,000, rate-limit anonymous clients. Pause non-critical bulk indexing (keep price/stock lane). Then diagnose: per-shard latency skew → a hot or merging shard; uniform → capacity. Add replicas only if nodes have headroom and relocation can be throttled.

Principal Approach: Peak readiness is a process: load tests at 4× last peak 6 weeks ahead, pre-scaled replicas, bulk-indexing freeze windows agreed with seller tools. The degradation ladder is pre-approved by the business with the expected conversion cost of each step (e.g., skipping L2 ≈ −1.5% conversion — far better than errors).

Key Takeaway: "Degrade ranking quality before availability. A worse order converts; an error page doesn't."

What clears the Staff bar:

• Degradation ladder before scaling
• Protects freshness of transactional fields during shedding
• Avoids relocation during peak

Deep Dive 2: Silent Relevance Decay#

Context: Over 6 weeks, search conversion dropped 7% with no deploys to the search cluster and no alerts. Leadership asks what happened.

Questions to Surface First:

• Is the drop uniform or concentrated in categories or query types?
• Did the zero-result rate or query-reformulation rate change?
• Did upstream data change (catalog fields, seller-provided titles)?
• Did the ranking model's feature distributions drift?

Typical L5 Approach: Retrain the ranking model on recent data. May bake the problem into the model if the cause is data quality.

Staff Approach: Slice by category and query class: the drop concentrates in apparel. Root cause: a catalog team changed brand from a normalized field to free text for new listings; brand filters and the brand_match feature silently degraded for ~30% of new apparel items. Fix the document contract (normalized brand via enrichment), backfill affected docs, add feature-distribution monitoring (null rate, cardinality per feature) and a schema contract check at the pipeline boundary.

Principal Approach: Search quality depends on upstream data it doesn't own. Establish document contracts with source teams (field semantics, not just types) enforced at pipeline ingestion, and a relevance health dashboard reviewed weekly by surface owners. Data changes that affect search features require search sign-off — like an API change.

Key Takeaway: "Ranking regressions are usually data regressions. Monitor features, not just outcomes."

What clears the Staff bar:

• Separates model from data with a holdout
• Monitors feature distributions
• Turns implicit dependencies into contracts

Deep Dive 3: Onboarding a Massive New Corpus#

Context: The company acquires a marketplace with 400M listings (8× current corpus) to be searchable within 3 months, in a unified search experience.

Questions to Surface First:

• Unified index or federated (separate indexes, merged results)?
• Do the listings share a schema and relevance features with ours?
• What's the update rate of the new corpus?
• What latency budget do we keep?

Typical L5 Approach: Reindex everything into the existing index with more shards. Changes shard count (requires reindex anyway), risks existing quality, and mixes very different ranking signals.

Staff Approach: Federate first: a separate index (acquired_v1, ~160 shards sized to 30 GB) with its own pipeline; the search API queries both in parallel and merges with a calibrated blending model (scores aren't comparable across indexes). Unify later if schemas converge. Capacity: 8× docs → roughly 8× storage and more query CPU; stage by category.

Principal Approach: Decide the long-term platform: one document contract and one ranking stack, or a federated architecture as a permanent feature (as LinkedIn does across verticals). Price both: federation costs a blending layer and ongoing calibration; unification costs a large migration. Set the decision point at 6 months with data.

Key Takeaway: "Federate to ship, unify when the data says so. Scores from different indexes aren't comparable — calibrate."

What clears the Staff bar:

• Federation vs unification as a staged decision
• Knows cross-index score calibration
• Protects existing quality with guardrails

Deep Dive 4: Post-Mortem — The Synonym Change That Emptied Search#

Context: A merchandiser updated the synonym file (index-time synonyms) and triggered an in-place reindex. A malformed rule mapped "apple" to a single-token pattern that broke tokenization for 2M products; for 3 hours, queries like "iphone case" returned zero results.

Questions to Surface First:

• Why could a merchandiser change index-time analysis without review?
• Why was the change applied in place instead of to a new index version?
• Why did zero-result rate not page?
• How fast was rollback possible?

Typical L5 Approach: Revert the synonym file and reindex again — another 3 hours of broken results.

Staff Approach: Immediate: if the previous index version still existed, swap the alias back (seconds). It didn't — so fall back to query-time synonyms disabled plus a restore from the last snapshot. Structural fix: synonyms move to query-time (search-time synonym filters, updatable without reindex); any analyzer change goes through versioned index + parity test; synonym edits via a tool with validation and a test-query suite; search.zero_result_rate > 5% for 5 min pages.

Principal Approach: Merchandising controls are a product with guardrails: a rules console with preview, validation, staged rollout and audit log, owned by the relevance team. Define which knobs are business-owned (boosts, synonyms within limits) and which are engineering-owned (analyzers, mappings).

Key Takeaway: "Anything that changes tokenization is a reindex; anything that's a reindex is a new version behind an alias."

What clears the Staff bar:

• Distinguishes query-time vs index-time config
• Rollback path via aliases and snapshots
• Business controls with guardrails, not bans

Deep Dive 5: Multi-Region Expansion#

Context: Expanding to EU and APAC with p99 ≤ 200 ms locally. EU seller data must stay in the EU; global products are searchable everywhere.

Questions to Surface First:

• Which documents are global vs region-restricted?
• Is ranking regional (local popularity, language)?
• How do index version swaps coordinate across regions?
• What's the failover story if a region's cluster fails?

Typical L5 Approach: One central cluster with a CDN in front. Cache misses cross oceans (~150 ms RTT) and blow the latency budget; residency is violated.

Staff Approach: Regional clusters fed by regional pipelines: global products replicated via Kafka to all regions; EU-restricted docs indexed only in EU. Regional ranking features (local popularity) computed regionally. Index version swaps rolled out region by region, starting with the smallest. Failover: route a region's traffic to the nearest region with degraded latency and without restricted docs — explicitly accepted by legal and product.

Principal Approach: A 3-region search platform roughly triples cluster cost (~$600K/month at this scale). Decide per region whether a full corpus is needed or a regional subset suffices; standardize the data-classification field on every document so residency is enforced by the pipeline, not per team.

Key Takeaway: "Search is derived, so regions can be independent. Residency is enforced in the pipeline, not the query."

What clears the Staff bar:

• Regional independence with classification-aware pipelines
• Region-by-region version rollouts
• Pre-approved failover semantics

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Size a document-partitioned index: shards, replicas and growth headroom, with fan-out tail controls
• Design a CDC-fed, versioned indexing pipeline with per-field freshness SLOs and end-to-end probes
• Reindex a live index via versioned indexes, dual-write, parity tests and alias swaps with rollback
• Build multi-stage ranking within a latency budget and evaluate changes offline and online
• Define a degradation ladder that trades ranking quality for availability
• Assign ownership across source teams, search platform and ranking owners

The Bar for This Question#

Mid-level (L4): Explains inverted indexes and TF-IDF/BM25; proposes Elasticsearch. Little on freshness, sharding math or ranking evaluation.

Senior (L5): Designs a working ES-based system with shards, replicas and an indexing pipeline; knows NRT refresh; adds popularity boosts. Reindexing and ranking evaluation handled superficially when asked.

Staff+ (L6): Treats the index as a rebuildable derived store behind aliases; sets per-field freshness SLOs with probes; reasons about fan-out tail latency and replicas vs shards; designs safe reindexing and gated ranking changes; assigns owners for documents, pipelines and relevance. The interviewer should learn something from the answer.

10. Staff Insiders: Controversial Opinions#

10.1 "Your Search Index Should Never Be a Source of Truth"#

The Staff position: Everything in the index must be reproducible from sources. If it isn't, you don't have a search engine — you have a fragile database.

Why this matters in interviews: "How would you rebuild it?" is the fastest way for an interviewer to find out whether the design is sound.

10.2 "Adding Shards Is Usually the Wrong Fix"#

Shards scale data; replicas scale QPS. Every added shard is one more request per query and one more chance to hit a slow node.

The Staff position: Right-size shards for data growth once per index version; scale reads with replicas, caching and cheaper ranking.

10.3 "Vector Search Doesn't Replace BM25"#

Lexical retrieval handles exact terms, SKUs, brand names and rare words better than embeddings; embeddings handle paraphrase and intent. Production systems that dropped lexical retrieval often reintroduced it.

The Staff position: Hybrid retrieval, with a re-ranker deciding. Add vectors when long-tail recall is the measured problem.

10.4 "Relevance Is an Org Problem Disguised as an ML Problem"#

The hardest ranking fights are between sponsored revenue, conversion, seller fairness and user trust. Models optimize what you tell them; someone has to own the objective.

The Staff position: Name the objective owner per surface and make business rules budgeted, measured and experimentally tested.

10.5 "Real-Time Indexing for Everything Is a Tax You Don't Need"#

Most fields don't need second-level freshness. Paying NRT cost for reviews and popularity burns capacity that should serve queries.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer builds a great product search. A Principal engineer sees seven search clusters across the company — product search, seller search, help center, internal docs, logs, support tooling, and a new semantic search prototype — each with its own pipeline, ranking approach and on-call, and three of them indexing the same catalog differently. The L7 question is what search capabilities the company provides as a platform, who owns relevance for each surface, and where standardization pays for itself.

The Org-Level Fault Line#

Central search platform vs surface-owned search stacks. A central platform (clusters, pipelines, index versioning, experiment and evaluation tooling) gives consistent operations and shared learning but can become a bottleneck and flatten surface-specific needs. Surface-owned stacks move fast but duplicate expensive infrastructure and produce inconsistent quality and security.

The L7 default: Centralize infrastructure and safety (clusters, pipelines, alias/versioning, ACL injection, evaluation and experiment tooling); decentralize ranking objectives and models to surface teams, within latency and experiment standards. Log search and analytics stay out — different access patterns and economics.

Cost Model#

Assumptions: cloud VMs with local NVMe at ~$1.2K/month per data node (16 vCPU/64 GB/2 TB), 3 copies of data, index ≈ 1.2× raw searchable text, engineer $25K/month fully loaded.

Relevance headcount usually exceeds infrastructure headcount at medium scale — and is where the revenue return lives. A 1% conversion lift on a $1B GMV search funnel is worth more than the entire cluster bill.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-SEARCH-002: Search Indexes and Relevance Changes

Scope: Every production search index serving user-facing or internal product surfaces (excluding log/observability search).

MUST:

• Index only data reproducible from a source of truth; full rebuild time documented and tested twice a year.
• Serve reads through aliases; mapping/analyzer changes ship as new index versions with parity testing and a ≥ 24-h rollback window.
• Carry a source version on every document; reject out-of-order writes.
• Publish per-field freshness SLOs and measure them with end-to-end probes.
• Build queries through the platform query library, which enforces tenant and ACL filters.
• Ship ranking changes only with an experiment readout including latency, zero-result-rate and business guardrails.

SHOULD:

• Size shards 10–50 GB; replicas for QPS with ≥ 30% headroom.
• Implement a degradation ladder with pre-approved steps.

Exceptions: Prototypes may skip experiments for ≤ 5% traffic for ≤ 30 days. Other exceptions reviewed by the Search Platform Council within 5 business days.

Success metrics: Zero ACL-leak incidents; median reindex (mapping change) < 1 day end to end; ≥ 90% of ranking launches with experiment readouts; duplicated catalog indexes reduced from 3 to 1 within a year.

What I'd Tell the VP#

"Search drives about 40% of our revenue, and today four teams run their own search systems, two of which index the same catalog differently. I recommend we consolidate the infrastructure — clusters, data pipelines, safety controls and experiment tooling — into one platform, while each product keeps control of its own ranking. That removes duplicate spend, closes the class of incidents where stale or leaked data showed up in search, and lets relevance improvements ship through a consistent testing process. It needs a platform team of about four engineers, funded partly by retiring two redundant clusters. The bigger return is relevance: every percentage point of search conversion is worth more than the entire infrastructure bill."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• A superb single-surface design with no view of the other six search stacks or shared catalog indexing.
• "We'll add vector search" without cost per query, recall targets, or who owns the embedding models across surfaces.
• A ranking experiment process without an owner for the objective when conversion and sponsored revenue conflict.

Appendices

term → postings: [(doc_id, tf, [positions]) ...]   sorted by doc_id, delta + block encoded
query "red running shoes": intersect postings(red) ∩ postings(running) ∩ postings(shoe)
                           using skip lists; score survivors; keep top-K heap per shard

score(q,d) = Σ_t IDF(t) · tf(t,d)·(k1+1) / (tf(t,d) + k1·(1 − b + b·|d|/avgdl))
k1 ≈ 1.2, b ≈ 0.75

{ product_id, version (source LSN), title (text + keyword), description (text, no positions if unused),
  brand_id (keyword), category_path (keyword, hierarchical), price (scaled_float), in_stock (bool),
  seller_id (keyword), popularity (float, nightly), region_class (keyword), acl? (keyword[]) }

Query:     search.qps, search.p50_ms, search.p99_ms, search.error_rate, search.partial_rate,
           search.cache_hit_rate, rerank.latency_ms, rerank.skipped_total
Index:     index.freshness_lag_s (probe), consumer_lag{partition}, index.bulk_rejections,
           index.merge_time_ms, index.segment_count, shard.size_gb
Quality:   search.zero_result_rate, search.ctr_at_10, search.conversion_rate, search.reformulation_rate,
           feature.null_rate{feature}
Cluster:   cluster.status, jvm.heap_pct, gc.pause_ms, shard.rejected_requests