Technologies referenced in this case study: PostgreSQL · ZooKeeper & etcd · Redis · DynamoDB · Cassandra · Apache Kafka

How to Use This Case Study#

Organized for interview use first, reference second. Read front-to-back once, then come back to individual sections for targeted review. Short codes for links are a different problem with different constraints — they live in URL Shortener. This case study is about the identifiers that become primary keys, public object IDs, ordering keys and shard routing keys.

Without a leased worker identity:
t=0:        Autoscaler adds 40 pods; worker_id is derived from hash(hostname) % 1024
t=+0s:      Two pods hash to worker_id 617 (birthday odds with 200 pods ≈ 18%)
t=+3min:    Both generate IDs in the same millisecond with the same sequence
t=+3min:    Order table INSERT on a unique PK fails → 0.02% of checkouts 500
t=+6 days:  Events table has no unique constraint → 11,400 duplicate event IDs
t=+6 days:  Downstream dedup drops 11,400 real events as "duplicates"
t=+9 days:  Finance reconciliation finds revenue gap. Root cause: two pods, one ID space.

With leased worker IDs + fencing:
t=0:        Autoscaler adds 40 pods; each acquires worker_id via etcd lease (TTL 30s)
t=+0s:      Lease grant is atomic — no two live pods can hold 617
t=+2h:      One pod is network-partitioned; lease expires at 30s
t=+2h:      Pod's own lease watchdog stops generation at 20s (10s safety margin)
t=+2h:      Pod returns 503 for ID requests until it re-acquires; callers retry another pod
Result:     Zero duplicates. One alert: id_gen.lease_lost_total += 1.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the intents and fault lines below.

What This Interview Actually Tests#

ID generation is not a bit-packing question. Everyone has seen the Snowflake diagram.

This is a uniqueness-without-coordination question that tests:

• Whether you ask what the ID is for before choosing a format
• Whether you know which assumption carries uniqueness (worker identity) and which carries ordering (the clock) — and what happens when each one breaks
• Whether you treat the ID format as a one-way door that outlives the service that generated it
• Whether you notice that an identifier is also an information channel to customers, competitors and attackers

The key insight: Uniqueness in a coordination-free generator is only ever as strong as the uniqueness of its worker identity, and ordering is only ever as good as its clock. Staff engineers spend their time on those two dependencies, not on the bit arithmetic.

The L5 → L6 → L7 Contrast — Start Here#

The Staff Positions#

The Four Intents#

Four intents. They pull in incompatible directions, which is why "one ID for everything" is the most common mistake.

🎯 Staff Move: "I'll design the internal primary key first — a 64-bit, k-sorted ID generated in-process with leased worker IDs — and treat the public ID as a separate opaque column. I'll be explicit about the ordering guarantee: per-generator monotonic, globally sorted only to within clock skew. If anyone needs strict global order, that's a sequencer, and I'd want to know why."

The Five Fault Lines#

In the Wild: Real Production Systems#

Why this section belongs here: Each of these is a publicly documented design that made a specific tradeoff. Citing the tradeoff is the signal, not the name.

Twitter — Snowflake#

Twitter announced Snowflake in 2010 when it moved off MySQL auto-increment. The layout is 1 unused sign bit, 41 bits of milliseconds since a custom epoch (~69 years), 10 bits of worker identity (datacenter + worker), and 12 bits of sequence (4,096 IDs per millisecond per worker). IDs are roughly time-ordered ("k-sorted"), not strictly ordered. When the IDs exceeded JavaScript's 2^53 safe integer range, Twitter's API added string variants of the ID fields (id_str) because JavaScript clients were silently rounding them.

Staff insight: Snowflake's two most-quoted lessons are not about bits — they are that k-sorted is a weaker guarantee than people assume, and that the wire encoding of a 64-bit ID is part of the format. Both are one-way doors.

Instagram — IDs Generated Inside PostgreSQL#

Instagram's 2012 engineering post describes generating IDs in PL/pgSQL inside each logical shard: 41 bits of milliseconds since a custom epoch, 13 bits of logical shard ID (8,192 logical shards mapped onto far fewer physical Postgres servers), and 10 bits of a per-shard sequence modulo 1,024. The ID therefore tells you which logical shard owns the row, with no lookup table.

Staff insight: Instagram embedded the logical shard, not the physical server — which is what lets logical shards move between machines without changing a single ID. That distinction is the difference between a routing key and a migration you will be forced into.

Meituan Leaf — Segment Mode and Snowflake Mode#

Meituan open-sourced Leaf, which offers two modes. Segment mode keeps a row per business tag with max_id and step; each server atomically bumps max_id by step, serves IDs from memory, and loads the next segment in the background before the current one runs out (double buffering), so the database sees one write per step IDs. Snowflake mode assigns worker IDs via ZooKeeper sequential nodes, caches them locally, and checks the local clock against the time recorded by peers, refusing to start if it has gone backwards.

Staff insight: Leaf is the clearest public example of treating the two hard dependencies — allocator database availability and clock sanity — as first-class design problems with explicit mitigations, rather than footnotes.

What Interviewers Probe#

System Architecture Overview#

Reading the diagram: The hot path is entirely in-process: no network call per ID. The control plane (etcd leases, format registry) is touched at boot and every ~10s for lease renewal — if it's unavailable, generators keep working until their lease's safety margin runs out. The segment allocator is an alternative for consumers that need dense IDs. The database's unique constraint and the db.duplicate_key_errors metric are the backstop that detects a uniqueness failure the generator didn't prevent.

Quick-Reference: The 30-Second Cheat Sheet#

Key Numbers Worth Memorizing#

Interview Walkthrough

The most common mistake: Candidates spend 20 minutes on bit arithmetic — "41 bits gives 69 years, 12 bits gives 4,096 per ms" — and never reach worker-ID leasing, clock regression, or public-ID leakage. The arithmetic takes 90 seconds. Spend the rest on the assumptions it rests on.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional requirement in one sentence:

"Generate identifiers that are unique across all machines and all time, at the rate our write path needs, without a single database sequence on the hot path."

Then spend the time on intent and the non-functional bar:

"Before I pick a format, four questions. Who sees the ID — only our databases, or customers too? Do we need IDs to sort by creation time, and how strictly? Does the ID need to tell us which shard owns the row? And what's the peak rate — thousands per second, or millions? I'll assume internal primary keys for a sharded OLTP store, peak ~500K IDs/s across ~300 service instances, rough time ordering for cursor pagination, and a separate public identifier for anything customer-facing."

Commit to the constraint set:

"So: 64-bit to stay in bigint and keep indexes small; generated in-process, no network hop; zero duplicates — enforced, not hoped for; k-sorted within clock skew, which I'll quantify; and the public ID is opaque."

🎯 Staff Move: Naming "public ID is a separate column" in the first two minutes heads off the enumeration probe and signals you've seen the incident. It costs one sentence.

Phase 2: Core Entities & API (1–2 minutes)#

• IdFormat: layout_version, epoch_ms, bit widths (ts, worker, seq), encoding (decimal string / base32)
• WorkerLease: worker_id, holder (pod UID), lease_id, expires_at, fencing_token
• Segment (only if dense IDs are needed): biz_tag, max_id, step, updated_at

The generator is a library, not a service:

IdGenerator.next() → int64                // hot path, ~100ns, in-process
IdGenerator.nextBatch(n) → int64[n]       // for bulk inserts
IdGenerator.parse(id) → {ts_ms, worker, seq, version}   // debugging only — not a contract
PublicId.mint(type) → "ord_7Fq2Kx9mW3pLzR8vN"            // 128 random bits, base62, prefixed

For callers that cannot embed the library (legacy stacks, SQL jobs):

POST /v1/ids:batch   { "count": 1000 }   →   { "ids": ["732...", ...], "layout_version": 1 }

🎯 Staff Move: "parse() exists for debugging, and I'd say in the API docs that the ID's internal structure is not a contract. The day a team starts using the embedded timestamp as their created_at is the day the layout becomes impossible to change."

Phase 3: High-Level Architecture (≤5 minutes)#

Walk it in 60 seconds:

1. On boot, the instance acquires a worker ID from etcd as a lease (worker/617 → pod-uid, TTL 30s). No lease, no IDs.
2. Each next() reads the clock, compares it to the last timestamp, increments the sequence or resets it, and packs the bits. No network.
3. The watchdog renews the lease every 10s. If renewal fails for 20s, the generator stops issuing IDs before the lease can be granted to someone else.
4. The database's unique constraint is the last line of defense, and its duplicate-key error rate is a monitored metric.
5. The public ID is minted independently with 128 bits of randomness and stored in its own uniquely indexed column.

🎯 Staff Move: "That's the design that works. Now the interesting part is the three assumptions it rests on: worker IDs are unique, the clock moves forward, and nobody depends on the ID's internals. Let me go through how each one fails."

Phase 4: Transition to Depth (1 minute)#

"The bits are the easy part. Three places I'd go deep: how worker identity is guaranteed unique under autoscaling and partitions, what the generator does when the clock goes backwards, and what the ID format commits us to for the next ten years — including what it leaks publicly. Which matters most for your system?"

If the interviewer has no preference, lead with worker identity — it is where silent duplicate IDs come from, and silent corruption outranks loud outages.

Phase 5: Deep Dives (25–30 minutes)#

Deep dive A — Worker identity (8–10 min). Walk the options: static config (duplicates on copy-paste), hash of hostname/IP (birthday collisions: 200 pods into 1,024 slots ≈ 18%), Kubernetes StatefulSet ordinal (unique per StatefulSet, not across clusters), consensus-store lease (the default). Explain the lease-expiry race and the self-fencing margin. Say the sentence: "A generator that cannot prove it still holds its worker ID must stop generating."

Deep dive B — Clock behavior (6–8 min). Clocks go backwards on NTP step corrections, VM live migration, leap seconds without smearing, and operator error. Policy: regression ≤ 5ms → spin until the clock catches up; regression > 5ms → either continue on a logical clock (max(now, last) with sequence overflow advancing logical time) or stop and page. Persist the last-issued timestamp so a restart after a backwards step doesn't reissue.

Deep dive C — Format as a one-way door (5–7 min). Epoch choice, bit budget, version bit, JSON string encoding, what consumers will do with the timestamp. Why logical shard bits are fine and physical shard bits are not.

Deep dive D — Public IDs and leakage (4–5 min). Enumeration, German-tank volume estimation, creation-time leakage, IDOR probing. Opaque prefixed IDs; authorization never relies on unguessability.

Deep dive E — If the interviewer wants dense IDs (4–5 min). Segment allocation with double buffering: DB write rate = ID rate / step, prefetch at 10–20% consumed, gaps on crash are acceptable, allocator DB outage tolerance = buffered IDs / consumption rate.

Phase 6: Wrap-Up (2–3 minutes)#

"To summarize: 64-bit k-sorted IDs generated in-process; uniqueness rests on leased worker IDs with self-fencing and a unique-constraint backstop; ordering rests on the clock, so we absorb small regressions and stop on large ones; public IDs are a separate opaque column. Day one, I'd ship the library and the lease service. I'd defer a network ID service, dense segment IDs, and multi-region bits until a consumer actually needs them — but I'd reserve two bits now so adding region or format version later doesn't require a migration."

🎯 Staff Move: Close with what you deferred and the bits you reserved. It shows you priced the one-way door.

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

ID generation is the smallest problem that contains every Staff-level theme. There is a single correctness property (uniqueness) that rests on an assumption the system does not control (worker identity). There is an ordering property that rests on another uncontrolled assumption (the clock). There is a storage-engine consequence (index locality) that a distributed-systems-only candidate misses. There is a security and privacy surface (what the ID leaks). And the format is a one-way door that will outlive every engineer in the room. A candidate who can find all five in a "simple" question will find them in a hard one.

1.2 The L5 → L6 → L7 Contrast — Visual#

The L5 path is a correct implementation of one intent. The L6 path interrogates the assumptions. The L7 path notices that the generator is cheap and the consumers are expensive, and governs the consumers.

1.3 The Staff Question That Cuts Through Everything#

"If two machines believe they are the same worker for ten seconds, what happens?"

Every design answers this question, explicitly or not. Snowflake without leases: duplicate IDs, detected (maybe) by a unique constraint. Snowflake with leases and self-fencing: impossible, at the cost of a generator that stops when it loses contact with the lease store. Segment allocation: impossible, because each range is handed out by an atomic DB update. UUIDv4/v7: irrelevant, because there is no worker identity — uniqueness rests on 74–122 bits of randomness instead. Ask this question and the design space sorts itself.

2. Problem Framing & Intent#

2.1 The Four Intents — Explained#

Intent 1: Database primary key. The ID is written into a clustered index and copied into every secondary index (InnoDB) or referenced by every foreign key. Size and insertion locality dominate (see Database Indexing). An 8-byte, time-ordered key appends to the right edge of the B-tree: pages fill to ~90%+, the hot pages stay in the buffer pool, and insert cost stays flat as the table grows. A 16-byte random key inserts into a random leaf: page splits leave pages 50–70% full, the working set becomes the entire index, and once the index exceeds memory every insert is a random read. The correctness bar is zero duplicates — but the enforcement is the unique constraint, and the generator's job is to make that constraint never fire.

Intent 2: Public / external identifier. The ID appears in URLs, receipts, support tickets, webhooks and customer databases. It must be stable forever (customers store it), opaque (it should reveal nothing), non-enumerable (adjacent IDs shouldn't exist), and ideally typed — ord_, cus_, inv_ prefixes make support tickets and logs self-describing and catch "passed a customer ID where an order ID was expected" bugs at the API boundary. The correctness bar is ≥ 96 bits of entropy and no reuse, ever.

Intent 3: Ordering / sortability. Feeds, timelines, audit logs and cursor pagination want IDs that sort by creation time, so WHERE id > :cursor ORDER BY id LIMIT 50 replaces a secondary index on created_at. The trap is assuming strictness. Snowflake IDs from two workers are ordered only to within the clock offset between them — an ID minted at 12:00:00.003 on worker A can be smaller than one minted at 12:00:00.001 on worker B if A's clock is 5ms behind. A cursor that reads "everything after ID X" can permanently skip rows that commit late with smaller IDs. Staff answer: define the guarantee (per-worker monotonic, global disorder ≤ ε) and design consumers to tolerate ε.

Intent 4: Sharding / routing key. Embedding a shard identifier lets any service route a read by inspecting the ID — no directory lookup, no extra hop. Instagram's 13 logical-shard bits are the canonical example. The trap is embedding physical topology: if the ID says "server 7," resharding means rewriting every ID or maintaining a translation table forever. Embed a logical shard from a large fixed space (4,096–8,192), map logical to physical in a small directory, and the mapping can change without touching any ID. Sharding & Partitioning and Database Sharding cover the directory side.

Committing:

🎯 Staff Move: "These four want different things: PKs want dense and ordered, public IDs want random and opaque, ordering wants a time prefix, and routing wants embedded location. I'm not going to find one ID that does all four well. I'll make the internal ID serve PK + ordering + logical routing, and mint a separate public ID. That's two columns and it removes an entire class of incidents."

2.2 When NOT to Build a Distributed ID Generator#

🎯 Staff Move: "If we have one Postgres primary doing 2K inserts a second, I wouldn't build anything — bigserial is the right answer and I'd revisit when we shard. The question gets interesting when a sequence becomes a cross-shard coordination point."

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. The Five Fault Lines#

3.1 Fault Line 1: Coordinated vs Coordination-Free#

The tension: Ticket servers and segment allocators get uniqueness from a database's atomic increment — no clock, no worker identity, dense IDs. They also make the database a dependency of every write. Snowflake and UUIDs remove the per-ID dependency but move the uniqueness guarantee to worker identity or to randomness.

Staff default: Snowflake-style with leased worker IDs for 64-bit PKs. Segment allocation when a consumer genuinely needs dense-ish integers (e.g., an external system with a 32-bit field, or a regulatory "invoice numbers must be sequential" requirement — and even then, per-tenant). UUIDv7 when 16 bytes is acceptable and you want zero control-plane dependency.

When to deviate: If the platform has no reliable consensus store, segment allocation is safer than Snowflake with hand-assigned worker IDs, because its uniqueness comes from a single atomic DB update you already know how to run.

🎯 Staff Move: "I'll pick coordination per-lease rather than per-ID. Snowflake moves the coordination from every write to one lease renewal every ten seconds — that's a 10^6 reduction in coordination traffic, and the lease store can be down for 20 seconds before anyone notices."

3.2 Fault Line 2: Sortable vs Opaque#

The tension: Time-ordered IDs make B-trees happy and cursors trivial. They also tell the world when each object was created, and — if sequential — how many objects you create.

Staff default: Internal IDs are time-ordered and never leave the trust boundary. Public IDs are opaque: 128 random bits base62-encoded with a type prefix, or the internal ID encrypted with a keyed permutation if you must avoid a second index (see Appendix B). Authorization never depends on an ID being unguessable — unguessability is defense in depth, not access control.

When to deviate: Some public IDs are meant to be sortable (tweet IDs, chat message IDs used as cursors by clients). That's fine if the time leakage is an accepted product decision with a named owner — Discord publicly documents that its snowflakes encode a timestamp.

3.3 Fault Line 3: Trust the Clock vs Don't#

The tension: A timestamp prefix gives ordering and makes worker-ID reuse safe across time. It also imports every clock failure into your uniqueness guarantee: if the clock goes back 3 seconds and the generator trusts it, it will reissue every (timestamp, sequence) pair from those 3 seconds.

How clocks go backwards in practice:

Options:

Staff default: Regression ≤ 5ms → spin. 5ms–1s → continue on a logical clock, increment id_gen.clock_regression_total, and alert if it persists > 60s. > 1s → stop issuing, mark the instance unhealthy so the load balancer drains it, and page. Persist the last-issued timestamp (to local disk every second, or to the lease record) so a process restarted after a backwards step resumes from max(now, persisted_last + safety_window).

When to deviate: If you use UUIDv7, the random bits make same-ms reissue harmless for uniqueness — the clock only affects ordering, so "trust it" becomes acceptable. That's a major reason to prefer UUIDv7 when 16 bytes is affordable.

🎯 Staff Move: "The clock policy needs an owner and a threshold, not a shrug. I'd rather have one instance refuse IDs and get drained than have it silently mint duplicates for three seconds — drained instances are a capacity blip, duplicates are a data-corruption incident."

3.4 Fault Line 4: 64-bit vs 128-bit#

The tension: 64 bits fits a native integer type in every database and language, halves index size, and keeps joins cheap — but 64 bits isn't enough room for randomness, so uniqueness needs worker coordination. 128 bits can be mostly random and coordination-free — at double the key size everywhere the key is copied.

Staff default: 64-bit for high-volume core tables where index size drives cost (events, messages, ledger lines). UUIDv7 for everything else — it removes an entire control plane. The decision is per-table, and that's fine.

3.5 Fault Line 5: Meaningful vs Meaningless IDs#

The tension: Every bit of meaning in an ID — shard, region, type, tenant — saves a lookup and costs flexibility. Once 10 billion stored references carry a meaning, that meaning cannot change.

Staff default: Embed only a logical shard (if routing matters) and a format version. Put type in the public ID prefix. Put region and tenant in columns. "Meaning in the ID is a cache of a lookup that can never be invalidated."

🎯 Staff Move: "I'll reserve 1 version bit and 2 region bits now and leave them zero. Reserved bits are cheap; a format migration costs quarters."

4. Failure Modes & Operational Reality#

ID generators fail in two shapes: loud (no IDs → writes fail → someone gets paged in 60 seconds) and silent (duplicate or misordered IDs → data quietly corrupts → someone finds it in a reconciliation three weeks later). A Staff design converts silent failures into loud ones, deliberately.

4.1 Duplicate Worker IDs — The Silent One#

t=0:        Deploy changes worker_id source from etcd lease to env var WORKER_ID
            (to "remove a dependency"); Helm chart templates it from the pod ordinal
t=0:        Two StatefulSets in two clusters both have ordinal 0..63 → 64 colliding pairs
t=+1min:    Collision requires same ms AND same sequence on both twins
            At ~2K IDs/s per pod, P(same ms, same seq) per ms ≈ low but nonzero
t=+1h:      orders table: 3 duplicate-key errors (unique PK) → 3 failed checkouts, retried OK
t=+1h:      events table: no unique constraint (append-only, "for speed") → duplicates land
t=+12 days: Analytics pipeline dedups by event_id → drops ~0.004% of real events
t=+19 days: Billing usage underreported by 0.004% across 2,100 customers. Found by finance.

Detection: db.duplicate_key_errors{table} > 0 is a page, not a warning — on a correct generator it is exactly zero. A sampled background job that checks (worker_id, ts) uniqueness across generators (id_gen.worker_lease_conflicts), and a startup check: every generator logs (worker_id, lease_id, holder) and an auditor alerts if two holders report the same worker ID within a TTL.

Blast radius: Every table written by the colliding pair. Tables without unique constraints absorb duplicates silently; downstream dedup turns duplicates into data loss.

Mitigation: Stop the affected generators (fail closed), re-lease, then run the duplicate audit on tables without unique constraints for the window since the change.

Prevention: Worker IDs only from leases; the env-var path does not exist in the library. Unique constraints on every table whose key is generated, including append-only ones — the 3–5% write cost is the price of detection. Contract test in CI: spin up two generators with the same config and assert the second refuses to start.

Owner: The ID platform team owns the library and lease service; each data-owning team owns its unique constraints. The post-mortem action item belongs to whoever approved removing the lease dependency.

4.2 Clock Goes Backwards#

t=0:        Cloud host performs live migration of 120 VMs; guest clocks pause ~1.8s
t=+2s:      chrony detects 1.8s offset; configured with makestep 1.0 -1 → steps clock back
t=+2s:      Generators on 120 instances see now < last by ~1,800ms
            Policy (old): spin-wait until now > last → each next() blocks up to 1.8s
t=+2s:      Request threads pile up; p99 write latency 12ms → 1,900ms
t=+5s:      Thread pools exhausted on 120 instances; health checks time out
t=+20s:     Load balancer ejects 120 instances (40% of fleet) at once; remaining 60% overload
t=+3min:    Partial outage; recovered as clocks pass "last" and instances re-admit

Detection: id_gen.clock_regression_ms (histogram), id_gen.wait_time_ms p99, host-level ntp.offset_ms and ntp.step_events_total.

Blast radius: Correlated — clock events are fleet-wide (same hypervisor, same NTP source, same leap second). A per-instance policy that is fine for one host becomes an outage across 40% of hosts.

Mitigation: Policy tiers: spin only up to 5ms; logical-clock continuation up to 1s; above 1s, refuse with a fast error (not a blocking wait) so the instance is drained cleanly rather than hanging. Configure NTP to slew, not step, after boot (makestep only in the first few updates), and smear leap seconds.

Prevention: Game day: inject a 2s backwards step on 10% of a canary fleet and verify drain-not-hang. A regression > 1s is an infrastructure incident — page the compute/time team, not just the service.

Owner: Time infrastructure (NTP/chrony config, leap smear) is owned by the compute platform. The generator's regression policy is owned by the ID library team. The post-mortem needs both in the room.

4.3 Allocator / Lease Store Unavailable#

Segment mode, step = 10,000, peak 50K IDs/s per biz_tag across 20 nodes:
t=0:        Allocator MySQL primary fails; failover takes 45s
t=0:        Each node holds current segment + prefetched next (double buffer)
            Per node: ~2.5K IDs/s; buffer ≈ up to 20,000 IDs ≈ 8s of headroom
t=+8s:      Nodes exhaust buffers → next() errors → order writes fail
t=+45s:     Failover completes; refills resume; 37s of write outage

Detection: id_gen.segment_remaining per node (alert when < 2× refill latency worth), id_gen.refill_latency_ms, id_gen.refill_errors_total.

Blast radius: Every writer of that biz_tag — usually an entire product line.

Mitigation & prevention: Size the buffer to survive the allocator's worst failover: step ≥ peak_rate_per_node × (failover_time × 2). Here: 2.5K/s × 90s ≈ 225K per segment. Dynamic step (Leaf adjusts step based on how quickly the last segment was consumed). Gaps on restart grow with step — fine, because nobody depends on density.

For Snowflake with leases, the equivalent is lease-store unavailability: with TTL 30s and a 10s safety margin, generators keep working for ~20s after etcd becomes unreachable, then stop. Mitigation is identical: TTL ≥ 2× the lease store's realistic failover time (etcd leader election is typically 1–3s, so 30s is generous) — and a new instance can't start without a lease, so a long lease-store outage blocks scale-out, not steady state.

Owner: The ID platform team owns allocator/lease-store availability and the buffer sizing formula. Product teams own their write-path behavior when next() fails (retry another instance, queue, or fail the request).

4.4 Sequence Exhaustion Under Burst#

t=0:        Bulk import job calls next() in a tight loop on one instance
t=0:        12-bit sequence → 4,096 IDs per ms → generator waits for next ms
t=0:        Import runs at 4.1M IDs/s — fine for the import
t=0:        Same instance also serves user writes; their next() calls queue behind the loop
t=+1s:      User-facing p99 for writes on that instance 8ms → 140ms

Detection: id_gen.seq_exhausted_total (counts ms rollovers due to exhaustion), lock-contention time on the generator mutex.

Mitigation: Bulk paths use nextBatch(n) on dedicated instances or a dedicated worker ID; generator uses a lock-free CAS loop on a packed (ts, seq) word so waiters don't convoy. If one logical stream legitimately needs > 4M IDs/s, give it multiple worker IDs — don't widen the sequence (that's a format change).

Owner: Calling team for workload isolation; ID library for lock-free implementation.

4.5 The Format Overflows Its Consumers#

Not a generator failure — a consumer failure, and the most common one in public APIs.

Owner: API governance (schema lint), ID platform (format docs), each consuming team.

4.6 Enumeration and Leakage Incident#

t=0:        Public invoice URL /invoices/1840211 (internal Snowflake-less sequence)
t=+1d:      Researcher increments ID, finds 3% of invoices load without auth (IDOR bug)
t=+1d:      Separately, analyst estimates monthly invoice volume from two IDs 30 days apart
t=+2d:      Disclosure; security incident; press asks about revenue numbers

Detection: api.404_rate_by_client spikes on sequential probing; WAF rule on monotonic ID scans per client.

Mitigation: Fix the authorization bug (the real vulnerability). Introduce opaque public IDs; keep the old URLs working behind auth with redirect-to-new-ID.

Owner: Security owns the IDOR fix; the product team owns migrating to public IDs; API governance owns the rule that blocks new endpoints from exposing internal keys.

4.7 Operational Reality Matrix#

5. Evaluation Rubric#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Reciting the Snowflake layout ≠ understanding ID generation. The layout is public; the lease and clock handling are the design.
• Knowing UUID versions by number ≠ judgment. The question is which property (order, opacity, size) you are trading and who pays.
• Collision-probability math ≠ uniqueness analysis. Random collisions at 122 bits never happen; operational collisions (cloned worker IDs, reissued timestamps) happen every year somewhere.
• "We'll use a ticket server like Flickr" ≠ dependency thinking — unless the candidate sizes batches and states the outage tolerance.

6. Interview Flow & Pivots#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Deriving bit math on the board — state it.
• Base62/base32 alphabet details — mention once.
• Exact NTP algorithm internals — name slew vs step, that's enough.
• Comparing every UUID version (v1, v3, v5, v6, v8) — v4 and v7 cover the decision.
• Short-code generation — point to URL Shortener.

6.4 Follow-Up Questions to Expect#

1. "How does a new instance get a worker ID, and what if two get the same one?"
2. "What does your generator do when the clock goes back 50ms? 5 seconds?"
3. "Your ID service is down — what happens to writes?" (Answer: there isn't one on the hot path; describe lease-store outage tolerance.)
4. "Can I use these IDs for cursor pagination? What breaks?"
5. "What does a competitor learn from your order IDs?"
6. "How would you change the epoch or add region bits after launch?"
7. "Why not UUIDv4 for everything?"

7. Active Drills#

Drill 1: The Opening#

Prompt: "Design a distributed ID generator."

Drill 2: Make the Layout Concrete#

Prompt: "Give me the exact bit layout and justify every field."

Drill 3: The Clock Goes Backwards#

Prompt: "NTP steps the clock back 800ms on one host. Walk me through what your generator does."

Drill 4: Worker IDs in Kubernetes#

Prompt: "You're on Kubernetes with HPA. How does each pod get a unique worker ID?"

Drill 5: Dependency Down — The Allocator#

Prompt: "You chose segment allocation for dense invoice numbers. The allocator DB is down for 2 minutes. What happens?"

Drill 6: The Hot Generator#

Prompt: "One ingestion service needs 20M IDs per second. Your layout gives 512K per worker."

Drill 7: Multi-Tenant Leakage#

Prompt: "We're a B2B SaaS. Enterprise customers can see their own object IDs. What's the risk?"

Drill 8: Build vs Buy#

Prompt: "Should we build an ID service, use a library, or just use UUIDv7 from the database?"

Drill 9: Changing the Format Without an Outage#

Prompt: "We used Unix epoch and 10 worker bits; we need region bits now. How do we change the format?"

Drill 10: Multi-Region#

Prompt: "We're going active-active in three regions. What changes?"

8. Deep Dive Scenarios#

Deep Dive 1: Peak-Traffic Incident — Generator Convoy During a Launch#

Context: During a product launch, write p99 on the order service jumps from 15ms to 900ms on 30% of instances. CPU is fine, the DB is fine. Profiling shows threads blocked inside IdGenerator.next(). You're escalated.

Questions to Surface First:

• Is the generator waiting on the clock (regression) or on the sequence (exhaustion) or on a lock?
• Did a bulk job or new feature start calling next() in a loop on the same instances?
• Are affected instances clustered on particular hosts (clock event) or particular deployments (code change)?

Typical L5 Approach: Adds more instances so load per instance drops, and widens the timeout. Latency improves but the root cause — a backfill job generating 4M IDs/s on shared request-serving instances, convoying behind a synchronized next() — remains, and returns at the next launch.

Staff Approach: Reads id_gen.seq_exhausted_total and lock-wait metrics: exhaustion is spiking on exactly the instances where the launch's backfill job is co-located. Moves the backfill to dedicated instances with their own worker IDs and nextBatch(), and ships the lock-free CAS implementation. Adds a guard: next() callers exceeding 1M IDs/s per instance get a log warning and a metric.

Principal Approach: Sees the incident as a workload-isolation failure, not a generator bug: batch and serving traffic share capacity with no isolation contract. Makes 'bulk ID consumers use dedicated worker-ID ranges and the batch API' part of the ID standard, adds it to the launch-readiness checklist, and has the platform team publish per-consumer ID-rate dashboards so the next team sees its own impact before launch.

Key Takeaway: "A generator that is fast on average can still convoy. Isolate bulk consumers and measure ID rate per consumer."

What clears the Staff bar:

• Uses generator-specific metrics to separate exhaustion from regression in minutes
• Fixes both workload placement and implementation
• Names the split ownership explicitly

Deep Dive 2: Silent Failure — Duplicate IDs Found Three Weeks Later#

Context: Data engineering reports that the events warehouse has 41,000 rows sharing an event_id with a different payload, all from the past 22 days. The events table in OLTP has no unique constraint. Nobody was paged.

Questions to Surface First:

• What changed 22 days ago in how worker IDs are assigned (deploy, new cluster, config)?
• Which worker_id values appear in duplicates, and which hosts/pods held them?
• Which downstream systems dedupe by event_id — and therefore dropped real data?

Typical L5 Approach: Finds that a new cluster was stood up 22 days ago with a copied config that hard-codes worker IDs 0–63, fixes the config, and writes a script to re-key the duplicate rows. Closes the incident.

Staff Approach: Fixes the config, then asks why the failure was silent. Three gaps: worker IDs could come from config at all; the events table had no unique constraint; no metric compares lease holders. Removes the config path from the library, adds the unique constraint (accepting ~3% write cost), adds a lease-conflict auditor, and quantifies downstream damage: every consumer that deduped by event_id lost data and must be backfilled from the raw log.

Principal Approach: Treats it as a class problem: any coordination-free generator with a non-leased identity can silently corrupt data, and there are likely other generators in the org with the same weakness (job schedulers, sharded counters). Commissions an inventory of every generator and its identity source, mandates leases from the shared identity service, and sets an org SLO — zero duplicate-key events, with each one reviewed like a security incident. Funds it by pricing the downstream reprocessing (weeks of data-engineering time) against the 0.5 FTE to run the shared lease service.

Key Takeaway: "The cost of a duplicate ID is paid by the consumer that dedupes it. Make duplicates loud at the source."

What clears the Staff bar:

• Asks 'why was it silent?' before 'what broke?'
• Traces downstream data loss caused by deduplication
• Removes the unsafe path rather than fixing the instance

Deep Dive 3: Large-Customer Onboarding — Importing 2 Billion Legacy Records#

Context: An acquired company's 2B records must be imported. Their records already have 64-bit IDs from their own Snowflake-like scheme with a different epoch, and their customers have those IDs in URLs and integrations.

Questions to Surface First:

• Do their customers reference these IDs externally? (Then they are public IDs and must be preserved verbatim.)
• Do their IDs overlap numerically with ours? Do they sort before or after ours?
• Will new records for those customers use our scheme or theirs?

Typical L5 Approach: Re-keys the 2B records with new IDs from our generator during import and stores the old ID in a column. Breaks every external integration that holds old IDs unless every lookup path is updated.

Staff Approach: Separates concerns: assign new internal IDs from our generator (via nextBatch on a dedicated worker range — 2B IDs at 512K/s per worker is ~65 minutes on one worker, or ~4 minutes on 16), and preserve legacy IDs as public identifiers in a legacy_id column with a unique index and a lookup path. External APIs accept both forms for those tenants. New records get our public ID format.

Principal Approach: Uses the acquisition to write the "foreign ID" chapter of the identifier standard: imported identifiers are always public aliases, never internal keys; alias resolution is a platform service; aliases have a documented retention policy. Plans the eventual sunset of legacy IDs with the acquired customers' contracts in mind — a one-way door owned by the business, not engineering.

Key Takeaway: "Imported IDs are public aliases, never internal keys."

What clears the Staff bar:

• Distinguishes internal and public identity during migration
• Sizes the batch generation with numbers
• Plans for long-lived alias ownership

Deep Dive 4: Post-Mortem — 40-Minute Write Outage From a Leap-Second-Style Clock Event#

Context: A time-source misconfiguration caused 35% of hosts to step their clocks back 1.2s simultaneously. Generators were configured to refuse IDs on any regression > 1s. Load balancers drained 35% of the fleet within 15s; the remainder overloaded; the write path was degraded for 40 minutes.

Questions to Surface First:

• Why did 35% of hosts receive the same bad time — shared NTP source? Same hypervisor pool?
• Was "refuse on > 1s" evaluated against correlated failure, or only single-host failure?
• Why 40 minutes — why didn't instances re-admit after the clock passed last_ts 1.2s later?

Typical L5 Approach: Raises the refusal threshold to 10s and moves on. This trades a loud outage for a larger silent ordering anomaly, and the next correlated event of 11s causes the same outage.

Staff Approach: Finds the 40-minute tail: instances that refused also failed readiness, and readiness only recovered on restart, which re-ran the "clock behind persisted last_ts" check — a refusal loop. Fixes: refusal is temporary and self-clearing once now > last_ts; for regressions between 1s and 10s, an instance cannot tell locally whether the event is isolated or fleet-wide, so it continues on the logical clock by default and pages; it refuses only above 10s. Fixes the NTP configuration to slew after boot.

Principal Approach: Treats it as a correlated-failure design gap: per-instance safety policies that are individually correct can be collectively catastrophic. Establishes a fleet clock-health signal (fraction of hosts with offset > 100ms) that every clock-sensitive system — ID generators, lease holders, token validators — consumes, and an org rule that safety policies triggered by shared infrastructure must degrade rather than drain. Adds time-source failure to the quarterly game-day calendar with the compute platform owning it.

Key Takeaway: "A safety check that drains instances is a correlated-failure amplifier when the cause is shared infrastructure."

What clears the Staff bar:

• Finds the reason for the long tail, not just the trigger
• Rejects 'raise the threshold' as the fix
• Distinguishes uniqueness preservation from ordering preservation under stress

Deep Dive 5: Multi-Region Expansion#

Context: The company is adding two regions. The current generator uses a single etcd cluster in us-east for worker leases and 10 worker bits with no region bits.

Questions to Surface First:

• Will regions be able to mint IDs while partitioned from us-east? (They must.)
• Do any consumers rely on global ID order across regions?
• How many generators per region at peak, including deploy overlap?

Typical L5 Approach: Keeps the single etcd in us-east; new regions acquire leases across the WAN. A transatlantic partition then blocks new instances in eu-west from starting and, after the safety margin, stops existing ones.

Staff Approach: Per-region lease stores with disjoint worker ranges (or the reserved region bits). No cross-region dependency for minting. Documents that global ordering degrades to cross-region skew (~50ms) and audits consumers that use ID order across regions — moving them to per-region cursors or to (created_at, id) with an overlap window.

Principal Approach: Ties ID topology to the org's regional cell strategy: each cell owns a slice of identity space, and the registry records allocations so a future sixth region doesn't collide. Notes that region bits encode minting location, and makes it a written rule that no compliance or residency decision may read them. Budgets the cross-region consumer audit as part of the multi-region program, not as an ID-team task.

Key Takeaway: "ID minting must survive a region partition. Partition the identity space, not the request path."

What clears the Staff bar:

• Eliminates cross-region dependency for minting
• Quantifies ordering degradation and audits consumers
• Separates minting location from data location

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Name the four ID intents and explain why internal and public IDs should be separate columns
• State, in one sentence, which assumption carries uniqueness and which carries ordering in any ID scheme
• Design worker-ID leasing with TTL, renewal and a self-fencing margin, and explain why hostname/IP derivation fails
• Give a tiered clock-regression policy with thresholds, and explain why per-instance policies can fail collectively
• Explain the B-tree consequence of random vs time-ordered keys, with page-fill and index-size numbers
• Define k-sorted precisely and design a cursor that tolerates it
• Describe what an ID leaks (time, volume, topology) and how opaque public IDs prevent it
• Identify epoch, layout, wire encoding and embedded topology as one-way doors, and plan a format migration
• Size segment allocation buffers against allocator failover time

The Bar for This Question#

Mid-level (L4): Knows UUIDs and auto-increment, maybe Snowflake by name. Can explain why a single auto-increment doesn't survive sharding. Doesn't reach worker identity or clock behavior without prompting.

Senior (L5): Produces a correct Snowflake layout with the arithmetic, mentions NTP and machine IDs, and may compare UUIDs. The design works on a whiteboard. Under probing, answers about worker-ID assignment and clock regression are improvised, and the public-exposure question is a surprise.

Staff+ (L6): Opens with intent and the internal/public split. Places the uniqueness guarantee in worker leases with self-fencing and a unique-constraint backstop, and has a clock policy with thresholds and an owner. Connects key choice to index locality with numbers, defines ordering precisely, names leakage, and treats the format as a one-way door with reserved bits. Spends under five minutes on bits. The interviewer should learn something from the answer — typically the correlated clock-failure case or the downstream-dedup-turns-duplicates-into-data-loss insight.

10. Staff Insiders: Controversial Opinions#

10.1 "You Probably Don't Need an ID Service"#

The Staff position: Ship a library and a lease mechanism. Offer a batch endpoint for stacks that can't embed the library. A synchronous per-ID network service is almost always a design smell.

Why this matters in interviews: Candidates who draw an "ID Service" box with replicas and a load balancer are solving availability for a problem they created. Saying "this is a library" saves five minutes and signals judgment.

10.2 "Auto-Increment Is Fine for Longer Than You Think"#

The Staff position: Use the database sequence until a sharding decision forces otherwise, but keep it out of public URLs from day one. The expensive mistake is not the sequence — it is exposing it.

Why this matters in interviews: Showing you'd not build something is a stronger signal than building it well.

10.3 "Never Expose Your Primary Key"#

The Staff position: Public IDs are a separate column, opaque, prefixed. The second unique index costs ~5% of write throughput and buys format freedom for the internal key forever.

Why this matters in interviews: It converts a privacy question into a data-modeling decision you already made.

10.4 "Strict Global Ordering Is a Requirement Smell"#

The Staff position: Ask which consumer needs strict global order and why. Usually the answer is per-key order (use a partitioned log) or "don't skip rows in pagination" (use an overlap).

Why this matters in interviews: Pushing back on a requirement with a cheaper, sufficient alternative is the Staff move.

10.5 "Embedding Physical Shard IDs Is a Year-3 Migration You Chose in Year 0"#

The Staff position: Logical shards or nothing. Topology belongs in a directory, not in billions of stored references.

Why this matters in interviews: It is the single best example of a one-way door hidden inside a "simple" bit layout.

11. The Principal Lens (L7)#

Why L7 Sees This Problem Differently#

A Staff engineer designs a generator. A Principal engineer finds that the org already runs six: a Flickr-style ticket server from 2014 that three services still call, two hand-rolled Snowflake variants with different epochs, UUIDv4 strings in the newer services, a Leaf-style allocator in payments, and a Mongo ObjectId leak into a public API. The generators are cheap; the consumers are the liability — mobile clients storing IDs as doubles, analytics jobs extracting timestamps from bits, partner integrations with 32-bit columns, support tooling that parses prefixes. At L7 the problem is identifier governance: one identity-lease primitive, a registry of formats, a public-ID standard, and a retirement plan for the legacy schemes.

🧭 Principal Move: "Before we design a generator, I'd like a list of every ID scheme in production and every system that parses one. My bet is the highest-leverage work is one lease service, one public-ID standard, and retiring the ticket server — not a new bit layout."

The Org-Level Fault Line#

Central ID service vs shared library vs per-team generators.

The deciding question: how many languages must the library support? If two, the library wins outright. If six, a sidecar or batch-endpoint fallback covers the long tail while the library covers the 80%.

Cost Model#

Assumptions: loaded engineer cost ~$25K/month; etcd cluster of 3 small nodes ~$300–$600/month per region; storage at ~$0.10/GB-month for primary SSD with 3× replication; buffer-pool memory at ~$4/GB-month.

The line that matters to leadership: The infrastructure is a rounding error. The money is in two places: storage — 8 extra bytes per key × 6 indexes × 50B rows ≈ 2.4 TB of index, roughly $1K/month in disk and far more in buffer-pool memory if it must be hot — and migrations: a format change touching 40 consumers costs 2–4 engineer-quarters (~$150K–$300K). Reserved bits and a registry are how you avoid paying the second one.

The 3-Year Evolution Path#

Each step is triggered by an event. Building per-region lease stores before the second region exists is wasted platform credibility.

One-Way Doors vs Two-Way Doors#

🧭 Principal Insight: Spend design review time in proportion to reversal cost. The generator implementation gets a code review. The epoch, layout, encoding and public prefixes get a design review with API governance, security and the largest consumers in the room.

The Standard I'd Write#

RFC: Identifier Standard (v1)

Scope: Every identifier persisted in a production datastore or exposed through any API, event or export.

Requirements:

• Internal identifiers MUST be generated by the platform library (64-bit layout v0) or as UUIDv7. New formats require registry approval.
• Generators with a worker identity MUST obtain it from the platform identity-lease service and MUST stop generating before lease expiry.
• Tables keyed by generated identifiers MUST have a unique constraint on the key; exceptions require director sign-off.
• Internal identifiers MUST NOT appear in external APIs, URLs, webhooks or exports. External identifiers MUST be opaque, ≥ 96 bits of entropy, base62, with a registered type prefix.
• 64-bit identifiers in JSON MUST be serialized as strings.
• Consumers SHOULD NOT parse internal identifiers; parse() is for debugging tools only.
• Generators MUST emit id_gen.clock_regression_ms, id_gen.lease_lost_total, and id_gen.seq_exhausted_total.

Exceptions: Filed with the API/data architecture group; decision within 5 business days; expiry ≤ 2 quarters.

Success metrics: Zero duplicate-key events per quarter; 100% of external APIs pass the ID lint within 3 quarters; legacy ID schemes reduced from 6 to 2 within 4 quarters; zero JSON precision bugs reported.

What I'd Tell the VP#

"We have six different ways of generating IDs, and two of them can silently create duplicates — we had one such incident last year that cost three weeks of data cleanup. I'm proposing one shared generator library, one service that guarantees each generator's identity is unique, and a rule that internal IDs never reach customers. It costs about one engineer for two quarters, then half an engineer to run. It removes a class of silent data-corruption incidents and stops leaking our order volume through public IDs. The biggest cost is migrating the old ticket server, which we'll do over three quarters with no customer-visible change."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll lease worker IDs from etcd." — correct, but silent on the other five generators in the org that don't.
• "Public IDs should be opaque." — correct, but no standard, no lint, no review gate, so the next team exposes its PK anyway.
• "We reserved two bits for the future." — good hygiene, but never priced the migration it avoids or named who allocates those bits.

Appendices

state: atomic int64 packed = (last_ts << SEQ_BITS) | seq

function next():
  loop:
    now = clock_ms() - EPOCH
    old = packed.load()
    last_ts = old >> SEQ_BITS
    seq     = old & SEQ_MASK
    if now < last_ts:
        regression = last_ts - now
        if regression <= 5:            spin; continue
        elif regression <= 10_000:     now = last_ts          # logical continuation
                                       metric clock_regression_ms.observe(regression)
        else:                          raise ClockRegressionRefused   # fast fail, drain
    if now == last_ts:
        if seq == SEQ_MASK:            now = last_ts + 1 if logical else wait_next_ms()
                                       new = (now << SEQ_BITS) | 0
        else:                          new = old + 1
    else:
        new = (now << SEQ_BITS) | 0
    if !lease.valid_with_margin():     raise LeaseLost
    if packed.compare_and_swap(old, new):
        ts, s = new >> SEQ_BITS, new & SEQ_MASK
        return (ts << TS_SHIFT) | (REGION << REGION_SHIFT) | (VERSION << VER_SHIFT)
               | (WORKER << WORKER_SHIFT) | s

-- allocator table
CREATE TABLE id_alloc (biz_tag VARCHAR(64) PRIMARY KEY, max_id BIGINT NOT NULL,
                       step INT NOT NULL, updated_at TIMESTAMP NOT NULL);

-- refill (one row lock, one round trip)
UPDATE id_alloc SET max_id = max_id + step, updated_at = now()
 WHERE biz_tag = :tag RETURNING max_id, step;
-- segment = [max_id - step + 1, max_id]

on next(tag):
  seg = current[tag]
  if seg.remaining < seg.size * 0.8 and next_seg[tag] is empty and not refilling:
      async refill → next_seg[tag]
  if seg.exhausted:
      if next_seg[tag] ready: current[tag] = next_seg[tag]; next_seg[tag] = empty
      else: wait up to 50ms for refill, then raise AllocatorUnavailable
  return seg.take()

CREATE TABLE tickets64 (id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
                        stub CHAR(1) NOT NULL DEFAULT '', UNIQUE KEY (stub)) ENGINE=InnoDB;
REPLACE INTO tickets64 (stub) VALUES ('a');
SELECT LAST_INSERT_ID();
-- two servers: auto_increment_increment = 2, offsets 1 and 2 → odd/even IDs

| 48 bits unix_ts_ms | 4 bits ver=0111 | 12 bits rand_a | 2 bits var=10 | 62 bits rand_b |

| 48 bits ms timestamp | 80 bits randomness |  → 26 chars Crockford base32, lexicographically sortable

Required lifetime L years      → ts_bits ≥ log2(L × 3.156e10 ms)      (50y → 41 bits)
Peak IDs/s per generator R     → seq_bits ≥ log2(R / 1000) + headroom  (500K/s → 9 bits)
Max concurrent generators G    → worker_bits ≥ log2(G × 2 for deploy overlap)
Reserved                       → version 1 bit, region 2 bits
Sign bit                       → 1 (keep positive in signed bigint)
Total must equal 64.

lease = etcd.grant(ttl=30)
for slot in shuffled(range(0, 1024)):          # shuffle to avoid thundering on slot 0
    ok = etcd.txn(
        compare=[create_revision("/idgen/workers/" + slot) == 0],
        success=[put("/idgen/workers/" + slot, pod_uid, lease=lease)])
    if ok: return slot, lease
raise WorkerSpaceExhausted   # alert at 80% occupancy, page at 95%

POST /v1/ids:batch
{ "count": 1000, "biz_tag": "orders" }
→ 200 { "ids": ["732410982735577088", ...], "layout_version": 0, "expires_hint_ms": null }
→ 503 { "error": "lease_unavailable", "retry_after_ms": 200 }

-- naive: may permanently skip rows that commit late with smaller IDs
SELECT * FROM events WHERE id > :cursor ORDER BY id LIMIT 100;

-- skew-tolerant: re-read an overlap window equal to the k-sorting bound plus commit latency
SELECT * FROM events
 WHERE id > :cursor - (:overlap_ms << TS_SHIFT)
 ORDER BY id LIMIT 100 + :overlap_estimate;
-- client dedupes by id; cursor advances to max(id) seen

# Uniqueness
db.duplicate_key_errors{table}             # must be 0 — page on any
id_gen.worker_lease_conflicts              # auditor: same worker_id, two holders
id_gen.generators_without_lease            # must be 0

# Clock
id_gen.clock_regression_ms (histogram)
id_gen.logical_clock_ahead_ms              # how far logical time leads wall clock
id_gen.refusals_total
fleet.clock_offset_hosts_over_100ms_pct

# Capacity
id_gen.seq_exhausted_total
id_gen.worker_space_occupancy_pct
id_gen.segment_remaining_seconds{biz_tag}
id_gen.next_latency_ns p99

extra_bytes = (16 − 8) × rows × (1 + secondary_indexes)          # clustered engines (InnoDB)
example     = 8 × 50e9 × 6 ≈ 2.4 TB of extra index