Technologies referenced in this case study: Apache Kafka · PostgreSQL · Redis · DynamoDB · Envoy, Kong & NGINX · OLAP Databases

Related: Notifications · Message Broker · Job Scheduler · Idempotency & Exactly-Once · Rate Limiter · Backpressure & Load Shedding · Push vs Poll · API Contracts That Age Well

Go deeper:

• Email delivery faces the same untrusted-receiver problem with an extra twist, since mailbox providers score your reputation and throttle accordingly; see Email Delivery Service.

Reading Guide#

Organized for interview use first, reference second. Read front-to-back once, then return to the fault lines and incidents that match your weak spots.

Without per-endpoint isolation:
t=0:        Customer A's endpoint starts responding in 29s (their DB is locked).
t=+30s:     Shared worker pool of 2,000: each A delivery holds a worker for 30s timeout.
            A produces 300 events/s → needs 9,000 worker-slots. Pool exhausted.
t=+1min:    Deliveries to all 40,000 other endpoints queue behind A. p99 latency: 2s → 14 min.
t=+5min:    Customers B–Z open tickets: "your webhooks are delayed". Status page goes amber.
t=+40min:   On-call manually pauses A. Backlog of 6M events drains over 25 minutes.

With per-endpoint concurrency limits and fair scheduling:
t=0:        Customer A's endpoint slows to 29s.
t=+30s:     A is capped at 20 concurrent deliveries; its queue grows, nobody else's does.
t=+2min:    A's circuit opens after 50% failures over 1 min → deliveries spaced by backoff.
t=+2min:    Other endpoints: p99 unchanged at 2s.
t=+3h:      A recovers. Its backlog drains at its own concurrency cap, oldest first.
            A's dashboard shows every attempt, status code and latency. Zero tickets from B–Z.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What the Interviewer Is Scoring#

A webhook platform is not a queueing question. Everyone can put events on Kafka and POST them.

It is a multi-tenant isolation and contract question that tests:

• Whether you design for the worst receiver, not the median one — slow, failing, malicious or misconfigured
• Whether your retry policy is a deliberate product promise with a defined end, not "retry with backoff"
• Whether you refuse to promise ordering you can't deliver — and know what to offer instead
• Whether customers can see and fix their own delivery problems without opening a ticket

The key insight: The platform's reliability is bounded by its least reliable receiver unless you isolate per endpoint. The queue is easy; the scheduler that keeps 40,000 independently failing endpoints from affecting each other is the system. Staff candidates design the scheduler; Senior candidates design the queue.

One Question, Three Levels#

Positions to Commit To#

Which Problem Are We Solving?#

Three intents produce three different systems. Name them, then commit.

🎯 Staff Move: "I'll design outbound webhooks to third-party customers — tens of thousands of endpoints we don't control. If the receivers were our own services, I'd use a broker with consumer groups and skip HTTP push entirely. If we're receiving webhooks, the design is a fast-ack ingestion buffer, which I can cover at the end. The interesting problems are isolation and the retry contract."

Where the Design Splits#

How Real Companies Built It#

Why this section belongs here: Three well-known webhook providers made visibly different choices about retries and redelivery. Naming the differences shows you see the retry policy as a product decision.

Stripe — Three Days of Retries, Signed Timestamps, No Ordering Promise#

Stripe's documentation states that in live mode it attempts delivery for up to three days with exponential backoff, that it does not guarantee events arrive in the order they were generated, and that endpoints may receive the same event more than once — receivers should log processed event IDs. Each delivery carries a Stripe-Signature header containing a timestamp and an HMAC-SHA256 signature over the timestamp and raw body; Stripe's libraries default to a 5-minute tolerance to reject replays, retries get a fresh timestamp and signature, and during secret rotation the old secret can stay valid for up to 24 hours with one signature generated per active secret. Manual resends are available for up to 15 days in the dashboard (Stripe docs).

Staff insight: Every one of those numbers is a contract — 3 days, 5 minutes, 24 hours, 15 days. In an interview, state your platform's equivalents explicitly; it shows you know the retry policy is something customers build against, not an internal tuning knob.

Shopify — Short Window, Fast Timeout, Subscriptions Removed on Persistent Failure#

Shopify's documentation says the receiving app must respond within five seconds or the delivery fails; failed deliveries are retried up to eight times over four hours, and if failures persist the subscription is removed, after which the app must recreate it and recover missed data from the outage period (Shopify docs).

Staff insight: A much shorter window than three days, paired with an aggressive response timeout, is a deliberate choice: it protects the platform's delivery capacity at the cost of pushing recovery onto receivers. Say: "Retry window length is a tradeoff between our infra and our customers' operational burden — and whatever we pick, receivers need a reconciliation path via the API."

GitHub — No Automatic Redelivery; Customers Redeliver Themselves#

GitHub's documentation states that it does not automatically redeliver failed webhook deliveries; instead it provides manual and API-driven redelivery, and its best-practices guide asks receivers to respond with a 2xx within 10 seconds. Each delivery carries an X-GitHub-Delivery header that is unique per event and preserved on redelivery, which receivers can use to detect duplicates (GitHub docs: failed deliveries, best practices).

Staff insight: "No automatic retries plus great redelivery tooling" is a legitimate design point — it puts control in the receiver's hands and keeps the platform's load predictable. The lesson for the interview: replay and an inspectable delivery log matter as much as the retry schedule, and a stable delivery ID is what makes redelivery safe.

Follow-Ups to Expect#

System Architecture Overview#

Reading the diagram: Producers write events to a durable log; the log is the replay source of truth. A matcher fans each event out to delivery tasks, one per subscribed endpoint. The scheduler is the heart of the system: it holds per-endpoint ready lists and a due-time index for retries, and only dispatches to endpoints with free concurrency slots and closed circuits, sharing capacity fairly across tenants. Workers sign and send through an egress proxy that blocks private address space. Every attempt lands in a customer-visible delivery log. The metric that tells you the platform is healthy is scheduler.oldest_ready_age for healthy endpoints — it should stay in seconds no matter how many endpoints are failing.

One-Minute Recap#

Numbers to Bring#

Interview Walkthrough

The most common mistake: Candidates spend 15 minutes on the event schema, the subscription API and Kafka partitioning, then run out of time before the interviewer asks the only question that matters: "One of your biggest customers' endpoints just started timing out. What happens to everyone else?" Compress the plumbing to ~8 minutes and spend the rest on isolation, the retry contract, security and customer-facing observability.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional scope in one breath:

"Customers register HTTPS endpoints and subscribe to event types. When an event happens in our platform, we POST a signed payload to every subscribed endpoint, retry on failure, and give customers a delivery log and replay. Receivers are third-party servers we don't control."

Then the non-functional requirements, which is where the design lives:

"Three constraints drive everything. One: isolation — at any moment a few percent of endpoints are slow or down, and they must not affect healthy endpoints' latency. Two: no silent loss — every event is either delivered or visibly failed after a documented retry window. Three: receivers are untrusted and our requests go to arbitrary URLs, so signing and egress security are requirements, not features. I'll assume 40,000 active endpoints across 10,000 tenants, 20K events per second average and 50K at peak, with a fan-out of about 1.4 deliveries per event."

Then name the underspecified parts:

"I'd confirm: do customers need ordering? How long should we retry? Are payloads allowed to contain personal data? I'll assume unordered by default with opt-in per-resource ordering, a three-day retry window, and thin payloads for sensitive event types."

🎯 Staff Move: Saying "at any moment a few percent of endpoints are down" reframes the problem from "deliver events" to "isolate failures" — that's the sentence that sets the level.

Phase 2: Core Entities & API (1–2 minutes)#

Name the nouns in 30 seconds:

• Event: event_id (globally unique), tenant_id, type, resource_id, resource_version, created_at, payload_ref
• Endpoint: endpoint_id, tenant_id, url, secrets[] (active + rotating), event_types[], max_concurrency, ordering (none / per_resource), state (active / circuit_open / disabled)
• Delivery: delivery_id, event_id, endpoint_id, attempt_count, next_attempt_at, state (pending / in_flight / delivered / failed_final), last_status
• Attempt: delivery_id, attempt_no, started_at, latency_ms, status_code, error_class, response_excerpt (first 1KB)

Customer-facing API:

POST   /v1/endpoints                       { url, event_types[], ordering? }
POST   /v1/endpoints/{id}/secrets:rotate   { expire_old_in: "24h" }
GET    /v1/endpoints/{id}/deliveries?status=failed&since=…
POST   /v1/deliveries/{id}/retry
POST   /v1/endpoints/{id}/replay           { since, until, event_types? }   → replay job
POST   /v1/endpoints/{id}/test             { event_type }

Delivery request (what the customer receives):

POST https://customer.example/hooks
Webhook-Id: evt_01HV…            (stable across retries)
Webhook-Timestamp: 1767225600
Webhook-Signature: v1=5257a8…    (HMAC-SHA256 of "timestamp.body"; one per active secret)
Content-Type: application/json
{ "id": "evt_01HV…", "type": "invoice.paid", "created": …, "data": { "object": { "id": "in_9", "version": 8, … } } }

🎯 Staff Move: "The event ID is stable across retries and replays; the signature and timestamp are fresh on every attempt. That combination lets receivers dedupe by ID while still rejecting captured-and-replayed requests by timestamp."

Phase 3: High-Level Architecture (≤5 minutes)#

Draw at most eight boxes:

Walk one event in 90 seconds:

1. Billing commits an invoice payment and an outbox row in one transaction; the relay publishes invoice.paid to the event log with event_id.
2. The matcher looks up subscriptions for (tenant, invoice.paid) — 2 endpoints — and creates 2 delivery tasks.
3. Each task lands on its endpoint's ready list. The scheduler picks endpoints round-robin within tenant shares, dispatching only if the endpoint has a free concurrency slot and its circuit is closed.
4. A worker signs the body with each active secret, sends through the egress proxy with a 10s timeout.
5. 2xx → delivered. Timeout, connection error, 5xx or 429 → schedule retry at now + backoff(attempt) ± jitter. Most 4xx → also retry (customers misconfigure and fix), but 410 Gone disables the endpoint.
6. Every attempt writes a row to the delivery log, visible to the customer within seconds.

🎯 Staff Move: Say out loud: "The scheduler never blocks on a slow endpoint — it skips endpoints without free slots. Worker capacity is only ever spent on endpoints that can accept work." You've now spent ~8 minutes.

Phase 4: Transition to Depth (1 minute)#

"That's the happy path, and it's the Senior-level design. What makes this hard is that thousands of endpoints are failing at any moment, customers want ordering we can't cheaply give, and we're making HTTP requests to arbitrary URLs. I'd like to go deep on isolation and scheduling, the retry contract and giving up, ordering, and security. Where would you like to start?"

If no preference: start with isolation. It's the question that decides the level.

Phase 5: Deep Dives (25–30 minutes)#

For each: state the tradeoff → commit → quantify → name who pays.

Deep dive 1: Isolation and fair scheduling (7–8 min)

"Each endpoint has a ready list and a concurrency cap, default 20. The scheduler keeps a set of 'dispatchable' endpoints — ready work, free slots, closed circuit — and picks among them with weighted fair sharing by tenant: each tenant gets a share proportional to plan, split among its endpoints. A slow endpoint fills its 20 slots and drops out of the dispatchable set; its queue grows, nobody else's does. Circuit breaker per endpoint: over 50% failures across at least 20 attempts in a minute opens it; while open, deliveries wait and one probe goes out every 30–60 seconds."

Quantify: "50K deliveries/s at 300ms median is ~15K in flight. A 30s endpoint capped at 20 slots consumes 20 of those, not 9,000. With async I/O, that's a handful of worker processes."

Who pays: "The failing customer's events get delayed — which is right; their endpoint is the cause. Healthy customers pay nothing."

Deep dive 2: The retry contract (6–7 min)

"Schedule: immediate, 5s, 30s, 2m, 10m, 30m, 1h, then every 2–3h until 3 days — about 16 attempts, each jittered ±20%. After the last, failed_final, visible in the log and replayable for 30 days. If an endpoint has had zero successes for 3 days, we disable it — emailing at first failure, at 24h and at disable. Disabling stops us retrying into the void and tells the customer something is structurally wrong."

Quantify: "With ~3% of endpoints failing at any time and their traffic share, retries add maybe 10–30% to attempt volume normally — and a burst when a large endpoint recovers, which the per-endpoint cap absorbs."

Deep dive 3: Ordering (4–5 min)

"Default unordered. Every event carries resource_id and resource_version; receivers keep the highest version seen per resource and drop older ones. Opt-in per-resource ordering: the scheduler dispatches only the oldest pending delivery per (endpoint, resource) — a failing event blocks that resource only, and the dashboard shows it as blocked with the reason."

Deep dive 4: Security (4–5 min)

"Signing: HMAC-SHA256 over timestamp.body, a header carrying one signature per active secret so rotation has an overlap window of up to 24 hours. Egress: every request goes through proxies that resolve DNS themselves, refuse private, loopback, link-local and metadata ranges — including after redirects, which we don't follow anyway — and send from a published IP range customers can allowlist."

Deep dive 5: Customer observability and ownership (3–4 min)

"Every attempt is visible to the customer within ~5 seconds: status, latency, error class, first 1KB of response. They can retry one delivery, replay a time range, or send a test event. The webhook platform team owns delivery health and pages on scheduler.oldest_ready_age for healthy endpoints. Customer endpoint failures are not our incidents — they're surfaced to the customer. Producing teams own event schemas and versions."

Phase 6: Wrap-Up (2–3 minutes)#

"The core idea: webhook reliability is bounded by the worst receiver unless you isolate per endpoint. So per-endpoint caps, circuits and a fair scheduler; a finite, documented retry contract with visible failure and replay; unordered delivery with versions instead of an ordering promise; and signing plus SSRF-safe egress because we're sending requests to arbitrary URLs."

The evolution closer:

"What I'd build later: dedicated delivery capacity for enterprise tenants; alternative destinations like delivering events into a customer's own queue or event bus for high-volume customers; and a schema registry with versioned event types. What I'd not build: exactly-once delivery — I'd invest in making receiver dedupe trivial instead."

🎯 Staff Move: End on who owns a failed delivery. "When an event fails after three days, the customer sees it, gets an email, and can replay it in one click. Our on-call never investigates a single customer's endpoint — that's what the delivery log is for."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Webhooks are the interview where your system's quality is graded by other people's code. You can't fix a customer's endpoint, you can't make it faster, and you can't stop it from returning 500 for three days. The candidate must design a system whose reliability for each tenant is independent of every other tenant's behavior — and whose failure modes are explained to customers by the product, not by your on-call. That is Staff work: designing blast-radius boundaries and operational contracts with parties you don't control.

It also has a deceptive cost structure. Healthy deliveries are cheap. Failing deliveries are expensive: they hold connections for the full timeout, they retry for days, they fill the delivery log, and they generate support tickets. A platform where 3% of endpoints fail can spend more than half its capacity on them unless the scheduler is designed to make failure cheap.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"Your largest customer's endpoint is returning 503 for every request and has been for six hours. They emit 2,000 events per second. Walk me through what your system is doing right now, what it costs, and what the customer sees."

A candidate who answers with the circuit state (open, probing every minute), the backlog (43M deliveries queued for that endpoint, retained, not consuming workers), the cost (storage, not compute), the customer view (dashboard banner, failure email at first failure and 24h, disable warning before day 3) and the recovery plan (drain at the endpoint's concurrency cap when probes succeed, oldest first or newest first by customer choice) has run a webhook platform. A candidate who says "we retry with backoff" has written a webhook sender.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Outbound webhooks to third-party customers → isolation and contract

• Constraint: untrusted, heterogeneous receivers; public, documented behavior; multi-tenant fairness
• Strategy: per-endpoint scheduling, finite retries, signing, SSRF-safe egress, delivery log, replay
• Failure mode: noisy-neighbor starvation, SSRF, silent drops, signature breakage during rotation
• Who pays for imperfection: healthy customers (delays caused by others), the platform's security team (SSRF), support (tickets)

Internal event fanout → use a broker

• Constraint: trusted consumers, high volume, need for replay and consumer-controlled pace
• Strategy: a log-based broker with consumer groups; consumers pull at their own rate
• Failure mode: building HTTP push with retries for consumers that could simply pull
• Who pays: the team maintaining a bespoke push system that duplicates the broker

Inbound webhook ingestion → fast-ack buffer

• Constraint: providers expect a 2xx within 5–10s; events arrive duplicated and unordered
• Strategy: verify signature, persist raw event keyed by provider event ID, return 200 in < 100ms, process asynchronously, reconcile against provider API
• Failure mode: processing inline and timing out → provider retries → duplicates; dropping events during deploys
• Who pays: the business, when a payment or order event is missed

2.2 When NOT to Build a Webhook Platform#

• Your receivers are your own services. Use a broker. Pull beats push when you control the consumer: backpressure is natural, replay is an offset reset, and there's no HTTP retry policy to design.
• You have fewer than ~100 endpoints and low volume. A job queue with a retry policy and a delivery table is enough. The per-endpoint scheduler earns its keep when noisy neighbors exist.
• Your customers need high throughput and strict order. Offer delivery into their own queue or event bus, or a pull-based events API with a cursor. Webhooks are the wrong transport for 5,000 ordered events per second to one consumer.
• Buying is viable. Managed webhook-sending services exist; if webhooks aren't core to your product, a vendor with per-endpoint isolation and a customer portal is likely cheaper than two engineers for a year — but the delivery contract you publish is still yours.

🎯 Staff Insight: "Webhooks are push over a network you don't control to code you don't control. I'd use them only when the receiver is external. For internal consumers, or high-volume external ones, a pull API with a cursor is simpler and more honest."

2.3 What the Interviewer Leaves Underspecified#

Interviewers deliberately omit:

• Ordering expectations — many candidates promise it; few can afford it
• Retry window and what happens after — "retry with backoff" isn't a policy until it ends
• Payload content — snapshot vs thin, and whether personal data may be pushed to customer URLs
• Endpoint validation — who checks that a URL isn't pointing at your internal network
• Customer visibility — whether customers can see attempts and replay, or must file tickets
• Peak shape — month-start billing runs or flash sales can produce 10–50× bursts for one tenant

Staff engineers surface these and commit. Senior engineers assume them away and get surprised by the follow-up.

2.4 Precise Terminology#

🎯 Staff Insight: If the interviewer says "guarantee delivery", ask: "Guarantee it within what window, and what does the customer see when the window closes? 'Guaranteed' without a window is a promise to retry forever."

3. Where the Design Splits#

Every webhook decision has a technical side (queue topology, retry math, signatures) and an organizational side (what we promise customers, who handles a failed delivery, who approves a retry-window change). Interviewers grade the second side.

3.1 Fault Line 1: Ordered vs Unordered Delivery#

The tension: Customers often ask for ordering — invoice.created before invoice.paid. Delivering in order requires serializing deliveries per key, which means one failing event blocks everything behind it, for as long as the retry window.

Staff default: "Unordered at-least-once is the documented default. Every event carries resource_id and a monotonic resource_version, and our docs show the five-line receiver pattern for ignoring stale versions. Customers who need order can opt into per-resource ordering per endpoint; the dashboard shows blocked resources and the event blocking them."

When to deviate:

• State-transfer events (the payload is the full current state): ordering barely matters — the latest version wins.
• Command-like events that can't be reconciled by version (rare): per-resource ordering opt-in, with a shorter retry window so blocks don't last days.
• High-volume ordered streams: steer the customer to a pull API with a cursor or delivery into their own queue.

🧭 Principal Move: "I'd make resource_version mandatory in the event schema standard for every producing team. Then ordering is never a platform guarantee — it's a property of the data, and every receiver can converge."

❌ Common L5 Trap: "We'll put each customer's events in one Kafka partition, so they're delivered in order." Order in the log is lost at the first HTTP retry or the first parallel worker. To keep it, you'd deliver one event at a time per customer and block on failures — at 200ms per delivery, 5 events/s per customer, and a three-day stall on the first bad event.

3.2 Fault Line 2: Shared Pool vs Per-Endpoint Isolation#

The tension: A single shared queue and worker pool is simple and efficient when all receivers behave. With 40,000 receivers, some always misbehave, and a shared pool lets the slowest one set everyone's latency.

Staff default: "Per-endpoint ready lists with a concurrency cap (default 20, plan-dependent), a circuit breaker per endpoint, and a scheduler that does weighted fair sharing across tenants — work-conserving, so idle capacity goes to whoever has work. Workers use async I/O so in-flight requests are cheap; a slow endpoint costs 20 sockets, not 20 threads."

When to deviate:

• Small platform (< 100 endpoints): a shared queue with per-endpoint in-flight counters in Redis is enough.
• Enterprise tenants with contractual latency: dedicated scheduler shards or worker pools, priced into the contract.

🧭 Principal Move: "Fairness by endpoint is gameable — a tenant registers 500 endpoints. Fairness by event is gameable — a tenant emits 1,000× more events. Fair share by tenant, weighted by plan, is the policy; I'd publish it so sales doesn't promise something the scheduler can't do."

❌ Common L5 Trap: "When deliveries slow down, we autoscale workers." Autoscaling on queue depth adds workers that the slow endpoint immediately consumes, because every new worker picks up the next item — which is likely to be for the endpoint with the biggest backlog. You've scaled the problem.

3.3 Fault Line 3: Retry Window and Giving Up#

The tension: Longer retry windows recover more events across longer customer outages but cost storage and attempts and delay the signal to the customer that something is broken. Shorter windows are cheap and push recovery onto customers.

Staff default: "About three days: attempts at 0, 5s, 30s, 2m, 10m, 30m, 1h, 2h, then every 3h — 16 or so attempts, ±20% jitter. Then failed_final, visible, replayable for 30 days. An endpoint with no 2xx for 3 consecutive days is disabled after two warning emails. Respect Retry-After on 429 and 503 up to a cap of 1 hour."

When to deviate:

• Time-sensitive events (OTP delivery, real-time alerts): short window — a 6-hour-old alert is worse than none. Product signs off.
• Financial events (payments, payouts): long window plus a reconciliation API, because a missed event costs money.

🧭 Principal Move: "The retry window is priced: attempts and storage on one side, support tickets on the other. I'd measure tickets per million failed_final deliveries for a quarter before changing the window, and publish a change with 90 days' notice — it's a contract."

❌ Common L5 Trap: "We retry 5 times and then put it on a dead-letter queue." Whose DLQ? If it's ours, nobody watches it. If it's invisible to the customer, the event is effectively dropped. The Staff answer makes final failure customer-visible and replayable — the DLQ is the customer's dashboard.

3.4 Fault Line 4: Thin vs Snapshot Payloads#

The tension: Snapshot payloads let receivers act without calling back — and are stale on arrival, large, and push potentially sensitive data to customer URLs. Thin payloads are small and always force a fresh read — and add API load and latency on the receiver side.

Staff default: "Hybrid, decided per event type in the schema. Small, non-sensitive types are snapshots with a resource version. Large or sensitive types — anything with personal data or exceeding ~64KB — are thin, and our API's read path is sized for the callback burst: if we emit 50K thin events/s at peak, assume up to 50K extra reads/s within seconds."

When to deviate:

• Regulated data (health, financial account numbers): thin only, always.
• Receivers on constrained networks or with no API credentials (simple integrations): snapshot.

🎯 Staff Insight: "Thin events convert webhook load into API load. If I switch a high-volume event type to thin, I'm telling the API team to absorb a synchronized read burst — that's a cross-team capacity conversation, not a payload format choice."

❌ Common L5 Trap: "Always send the full object so receivers don't have to call back." Then a retried subscription.updated from an hour ago arrives after a newer one and overwrites the receiver's current state with stale data — and the payload contains a customer's address sitting in a third party's request logs.

3.5 Fault Line 5: Trust Boundary — Signing and Egress#

The tension: Receivers must verify that a request came from you; you must ensure that a customer-supplied URL can't make your infrastructure attack itself. Both are security requirements that interact with operations: secret rotation must not drop deliveries, and egress controls must not break legitimate endpoints.

Staff default: "HMAC-SHA256 per endpoint, signing timestamp.body, a signature header with one entry per active secret so rotation has a configurable overlap up to 24 hours. Egress only through proxies in an isolated network segment with no route to internal services; the proxy resolves DNS itself, pins the IP for the connection, blocks private and metadata ranges and doesn't follow redirects. We publish our egress IP ranges so customers can allowlist."

When to deviate:

• Regulated customers may require mTLS — offer it as an enterprise option.
• A single public key for all receivers can be preferable when customers dislike secret handling, at the cost of coordinated key rotation.

🧭 Principal Move: "Customer-supplied URLs are an attack surface that the webhook team will under-invest in because it's 'just HTTP'. I'd have security own the egress proxy's threat model and pen-test it twice a year, with DNS-rebinding and redirect cases in the test suite."

❌ Common L5 Trap: "We validate the URL when the customer registers it — we reject private IPs." DNS can change after validation (rebinding): the hostname resolved to a public IP at registration and resolves to 169.254.169.254 at delivery time. The check must happen at connect time on the resolved IP, every time.

4. When It Breaks#

4.1 The Slow Giant — One Endpoint Eats the Pool#

t=0:       Tenant T7 (12% of all events) deploys a handler that holds a DB lock.
           Endpoint latency: 120ms → 29s. No errors yet — just slow 200s.
t=+30s:    Shared pool (pre-isolation design): T7 deliveries occupy 3,600 of 4,000 slots.
t=+1min:   scheduler.oldest_ready_age for healthy endpoints: 2s → 90s.
t=+3min:   Autoscaler adds 50% workers. T7 absorbs them within 40s.
t=+6min:   Page: delivery.first_attempt_p99{healthy} > 60s for 5m.
t=+9min:   On-call sets T7 endpoint max_concurrency=20 by hand. Healthy p99 recovers in 2 min.
t=+4h:     T7 fixes handler. Its 21M-delivery backlog drains at 20 in-flight over ~6h.

Detection: delivery.first_attempt_p99 split by endpoint health; endpoint.inflight{endpoint} top-N; worker.slots_used_by_top_endpoint_pct.

Mitigation: enforce per-endpoint concurrency caps (default, not opt-in); latency-aware circuit — slow successes count against the endpoint too once p99 exceeds 10s.

Prevention: caps on by default for every endpoint; load test with 5% of endpoints at 30s latency.

Owner: webhook platform on-call.

4.2 The Recovery Stampede — 3 Days of Backlog in 3 Minutes#

t=0:       Tenant T3's endpoint (load balancer misconfig) has failed for 60 hours.
           Backlog: 9.4M deliveries in RETRY_WAIT, all due within the next hour.
t=+0:      T3 fixes the LB. Next probe succeeds. Circuit closes.
t=+10s:    Scheduler dispatches T3 at cap 200 (enterprise plan). T3's handler does 40/s.
t=+40s:    T3's servers fall over again. 502s. Circuit reopens.
t=+2min:   Cycle repeats every ~3 minutes. T3 opens a Sev-1 ticket against us.

Detection: endpoint.circuit_flaps{endpoint} (open→closed→open within 10 min); per-endpoint success rate after close.

Mitigation: slow-start on circuit close — begin at 2 in-flight, double every 30s while success rate > 95% (like TCP slow start); customer-configurable max delivery rate.

Prevention: slow-start is the default recovery behavior; dashboard lets customers choose drain order (oldest-first or newest-first) and pause/resume.

Owner: webhook platform.

4.3 The Silent 200 — "You Never Sent It"#

t=0:       A customer's new framework middleware returns 200 for every webhook
           before their handler runs, then the handler throws.
t=+5 days: Customer: "40,000 order events are missing. Your platform dropped them."
t=+5 days: Support has no visibility into customer handlers; escalates to engineering.
t=+5 days: Delivery log shows 40,000 attempts, all 200 in ~4ms, from our egress IPs.

Detection: this is the customer's failure — the platform's job is evidence. Delivery log with timestamps, status codes, response latency and response-body excerpts. A latency anomaly (p50 4ms vs historical 180ms) can be surfaced to the customer as a hint.

Mitigation: customer replays the window from the dashboard.

Prevention: customer-facing delivery log, self-serve replay, and documentation recommending receivers persist before acknowledging.

Owner: customer (root cause); platform owns the evidence and replay.

4.4 SSRF — The Metadata Endpoint#

t=0:       Attacker registers https://hooks.attacker.example as an endpoint. Resolves public.
t=+1h:     Attacker changes DNS: hooks.attacker.example → 169.254.169.254, TTL 1s.
t=+1h:     Worker (direct egress) resolves at send time, connects to cloud metadata service.
t=+1h:     Response excerpt (first 1KB) shows instance credentials in the attacker's delivery log.

Detection: egress.blocked_destination_total (should be nonzero — attempts happen); alerts on any successful connection to a non-public range (should be impossible).

Mitigation: rotate exposed credentials; disable the endpoint and tenant.

Prevention: egress proxy in an isolated segment; resolve-once-and-pin; block private, link-local, loopback, metadata and own VPC ranges at connect time; no redirects; instance metadata requires session tokens; response excerpts never shown for blocked or non-public destinations.

Owner: security (threat model), webhook platform (proxy).

4.5 Duplicate Storm After a Scheduler Failover#

t=0:       Scheduler primary loses its lease. In-flight state for 14K deliveries not yet persisted.
t=+15s:    Standby takes over; treats those 14K as pending; re-dispatches.
t=+16s:    Customers receive ~14K duplicates within seconds. Most dedupe by Webhook-Id.
t=+2h:     Two customers without dedupe report double-fulfilled orders.

Detection: delivery.redispatch_after_failover_total; customer reports.

Mitigation: this is at-least-once working as designed; the event ID made it safe for receivers that follow the docs.

Prevention: persist IN_FLIGHT with a lease before dispatch so the standby only re-dispatches expired leases; keep the docs' "dedupe by ID" guidance prominent with code samples.

Owner: webhook platform (scheduler), customers (dedupe).

4.6 Secret Rotation Breaks Verification#

A customer rotates their signing secret with "expire old immediately", but their receiver fleet takes 20 minutes to roll out the new secret. Every delivery in that window fails verification — the receiver returns 401, we retry, and the backlog clears once the rollout finishes. If they had chosen the 24-hour overlap, both signatures would have been in the header.

Detection: per-endpoint spike in 401/403 right after a rotation event — surface it in the dashboard with a hint.

Prevention: default overlap of 24 hours; dashboard warns when choosing immediate expiry.

Owner: customer (rollout), platform (safe default).

4.7 Operational Reality Matrix#

🎯 Staff Insight: The platform's health metric is healthy-endpoint latency, not overall latency. "Overall p99 will always look bad, because some endpoints are always down. I'd page on first-attempt p99 for endpoints whose last 20 attempts succeeded — that's the number that only moves when we're the problem."

5. Scorecard#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• Deep Kafka knowledge ≠ webhook design. Partition counts and consumer-group tuning don't address the HTTP hop, where every interesting failure lives.
• Elaborate retry math ≠ a retry contract. A beautiful backoff formula without a stated end, final state and customer notification is incomplete.
• "We use a circuit breaker" ≠ isolation. A global circuit breaker on "the webhook service" protects nothing; it must be per endpoint.
• Security buzzwords ≠ SSRF defense. "We use a WAF" doesn't stop your own workers from calling the metadata service.

6. The 45 Minutes, Phase by Phase#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Kafka internals — "the log is durable and partitioned by tenant; delivery state lives in the scheduler."
• Subscription CRUD — one line.
• Event schema design per product — producers own it; mention versions.
• Dashboard UI — say what it shows, not how it looks.
• Exactly-once — say why you won't offer it, in one sentence.

6.4 Follow-Up Questions to Expect#

1. "One endpoint takes 30 seconds per request and gets 12% of all events. Walk me through the next 10 minutes."
2. "How long do you retry, what's the schedule, and what happens at the end?"
3. "A customer insists on ordering. What do you offer, and what does it cost them?"
4. "How does a customer rotate their secret without dropping deliveries?"
5. "Show me how a customer-registered URL could reach your internal network, and how you stop it."
6. "An endpoint recovers after 3 days with 9M queued deliveries. What happens when the circuit closes?"
7. "How big is the delivery log, and how do customers query it?"

7. Practice Rounds#

Drill 1: The Opening#

Prompt: "Design a webhook system for our platform."

Drill 2: The Slow Endpoint#

Prompt: "Your largest customer's endpoint starts taking 29 seconds to respond. They're 12% of traffic. What happens?"

Drill 3: Make It Concrete — Size the Workers and the Log#

Prompt: "Size the delivery tier and the delivery log for 50K events/s peak."

Drill 4: The Retry Contract#

Prompt: "Define your retry policy. All of it."

Drill 5: The Recovering Giant#

Prompt: "A customer's endpoint was down for 3 days. It just came back. 9 million deliveries are waiting. What happens?"

Drill 6: The Noisy Tenant#

Prompt: "One tenant emits 40% of all events during their billing run on the 1st. Other tenants complain about delays. Fix it."

Drill 7: Build vs Buy#

Prompt: "Should we build this or use a managed webhook-sending service?"

Drill 8: Changing the Retry Policy Without an Outage#

Prompt: "Finance wants to cut the retry window from 3 days to 24 hours to save cost. How do you ship it?"

Drill 9: The Cost of Failure#

Prompt: "Where does the money go in this system, and how would you cut it by 30%?"

Drill 10: Multi-Region#

Prompt: "We're going multi-region. How do webhooks work?"

8. Incident Walkthroughs#

Deep Dive 1: Peak-Traffic Incident — First-of-Month Billing Run#

Context: At 00:00 UTC on the 1st, event volume jumps from 20K/s to 140K/s as subscription renewals fire across thousands of tenants. Healthy-endpoint first-attempt p99 goes from 2s to 11 minutes. The on-call escalates to you.

Questions to Surface First:

• Is the delay in ingest (event log lag), fan-out (matcher lag) or dispatch (scheduler ready age)?
• Are workers saturated, or is the scheduler not dispatching?
• Is one tenant dominating, or is it broad?
• Did any endpoint group start failing at the same time (customers' own billing handlers overloaded)?

Typical L5 Approach: Doubles workers. Matcher lag was the bottleneck, so nothing improves for 20 minutes.

Staff Approach: Reads the pipeline stage by stage, finds matcher consumer lag at 9 minutes because subscription lookups miss the cache under the burst, scales the matcher and pre-warms the subscription cache. Confirms the scheduler's tenant fair share kept small tenants' latency bounded once events reached it.

Principal Approach: Treats the 1st of the month as a known, scheduled peak and turns it into a capacity plan: pre-scaling on a calendar, load tests at 10× baseline before each month-end, and a conversation with billing about spreading renewals across the first hour.

Key Takeaway: "Find the stage that's slow before adding capacity. Webhook pipelines have three queues, and only one of them is usually the problem."

What clears the Staff bar:

• Localizes the bottleneck to a pipeline stage before acting
• Fixes the cache behavior, not just capacity
• Turns a predictable peak into a plan with the producing team

Deep Dive 2: Silent Failure — Deliveries "Succeeded" With Nothing Sent#

Context: A large customer reports that no order.shipped events have arrived for 36 hours. Your dashboard shows all deliveries to their endpoint as delivered with 200s.

Questions to Surface First:

• Are 200s coming from the customer's server, or from something in between?
• Did anything change in our egress path — proxy, DNS, TLS?
• What do the response excerpts and latencies look like compared to last week?
• Is this one customer or every customer behind a particular proxy?

Typical L5 Approach: Tells the customer the log shows 200s, so the problem is on their side.

Staff Approach: Notices latency dropped from 180ms to 3ms and the response excerpt is an HTML captive page. A proxy config change routed a subset of destinations through a corporate filtering service that returns 200 with a block page. Rolls back, replays 36 hours for affected endpoints.

Principal Approach: Establishes that "200" isn't sufficient evidence of delivery when the path includes infrastructure we operate. Adds end-to-end canaries — synthetic endpoints that verify signatures and report receipt — and treats egress as a production dependency with change review.

Key Takeaway: "A 200 is evidence only if you know who sent it. Canary receivers are how you know."

What clears the Staff bar:

• Uses latency and response excerpts as forensic evidence
• Doesn't reflexively blame the customer
• Adds end-to-end verification, not just more logging

Deep Dive 3: Large-Customer Onboarding — 15,000 Events/s to One Endpoint#

Context: A new enterprise customer wants all events — about 15,000 per second at peak — delivered to a single endpoint behind their API gateway, in order per account. Sales has signed. You're asked to make it work.

Questions to Surface First:

• What can their endpoint actually sustain? Latency at that rate?
• Do they need order per account, or will resource versions suffice?
• Would a different transport — delivery into their queue, or a pull API — serve them better?
• What's the blast radius on our shared capacity?

Typical L5 Approach: Raises their endpoint cap to 5,000 and enables strict ordering. Ordering serializes per account, throughput collapses on hot accounts, and the cap consumes shared capacity.

Staff Approach: Measures their endpoint: 300ms p50 at 2,000 concurrent. Proposes per-resource ordering (account-level) with a cap of ~5,000, on dedicated scheduler and worker capacity, plus batching — up to 100 events per request — which cuts request count 50–100×. Offers delivery into their own queue as the long-term option.

Principal Approach: Recognizes a product gap: high-volume customers need a different delivery product. Adds batched delivery and queue destinations as an enterprise tier, priced on reserved capacity.

Key Takeaway: "When one customer needs 15K/s in order, webhooks are the wrong transport unless you batch — and they probably want a queue."

What clears the Staff bar:

• Measures the receiver before committing
• Uses batching and dedicated capacity instead of raising shared caps
• Offers a better-suited transport

Deep Dive 4: Post-Mortem — Credentials Exposed via a Customer Endpoint#

Context: A security researcher reports that registering an endpoint with a hostname that later resolves to a link-local address returned cloud instance metadata in the delivery log's response excerpt. You're leading the post-mortem.

Questions to Surface First:

• Which egress path did these requests take? Why wasn't the proxy blocking them?
• Were credentials actually exposed? Which roles, which permissions?
• How many endpoints have ever resolved to non-public ranges?
• Do workers have any route to internal networks?

Typical L5 Approach: Adds a check at registration that rejects private IPs.

Staff Approach: Finds that a retry path bypassed the proxy for "faster retries" and resolved DNS at send time. Rotates credentials, removes the bypass, moves IP checks to connect time on the pinned resolved address, and hides response excerpts for any non-public destination.

Principal Approach: Treats egress as a security boundary owned by security, with network-level enforcement — workers in a segment with no route except via the proxy — so a code path can't bypass it. Adds the scenario to the pen-test suite and requires session tokens for instance metadata everywhere.

Key Takeaway: "SSRF defenses in application code get bypassed by the next optimization. Enforce the boundary in the network."

What clears the Staff bar:

• Recognizes registration-time checks are bypassed by DNS rebinding
• Moves enforcement to connect time and to the network layer
• Hides excerpts for non-public destinations

Deep Dive 5: Multi-Region Expansion — EU Data Residency#

Context: The company is launching EU data residency. EU tenants' data must not leave the EU, including event payloads and delivery logs. Webhooks currently deliver from one US region.

Questions to Surface First:

• Where are EU tenants' events produced? Already in an EU region?
• Do delivery logs and response excerpts count as customer data? (Yes — excerpts can contain anything.)
• Will egress IPs change, and how do customers with allowlists find out?
• How are cross-region tenants (US company, EU subsidiary) handled?

Typical L5 Approach: Deploys a second copy of the webhook stack in the EU and routes EU tenants there. Forgets the delivery log is replicated to the US analytics warehouse.

Staff Approach: Region-local everything for EU tenants: event log, scheduler, workers, egress, delivery log and replay storage. Analytics gets aggregated metrics only. New EU egress IPs are published 90 days ahead; tenants migrate with a dual-region delivery window where both IP ranges are valid.

Principal Approach: Defines residency as a tenant attribute honored by every platform that touches customer data, with an audit that proves no EU payload or excerpt exists outside the EU. Webhooks is one of many systems; the standard is company-wide.

Key Takeaway: "Residency includes the logs. Response excerpts are customer data you didn't know you were storing."

What clears the Staff bar:

• Includes delivery logs and excerpts in the residency scope
• Plans egress IP changes as a customer migration
• Proves residency with tests, not assertions

9. Level Expectations Summary#

After studying this case study, you should be able to:

• Explain why webhook reliability is bounded by the worst receiver without per-endpoint isolation
• Design a scheduler with per-endpoint concurrency caps, circuit breakers (error- and latency-aware) and tenant fair share
• State a complete retry contract: classification, schedule, jitter, final state, disable rule, notifications and replay
• Choose unordered delivery with resource versions, and price opt-in per-resource ordering
• Decide thin vs snapshot payloads per event type, including the API load thin events create
• Sign deliveries with HMAC over timestamp and body, support overlapping secrets during rotation, and enforce SSRF defenses at connect time
• Size workers by concurrent requests (async), and size the delivery log as the largest storage cost
• Handle recovery from long outages with slow-start and customer-controlled drain

The Bar for This Question#

Mid-level (L4): Builds a queue and workers that POST events with retries. Works for well-behaved receivers. No isolation, no end to retries, no signing beyond HTTPS.

Senior (L5): Adds exponential backoff, a DLQ, HMAC signing, Kafka partitioning and autoscaling. The gap: shared worker pool lets one slow endpoint stall everyone; promises ordering via partitions; retries end in an unowned DLQ; registration-time URL checks only. The design works in a demo and fails on the first slow giant.

Staff+ (L6): Frames the problem as isolation against untrusted receivers within the first five minutes. Designs per-endpoint caps, circuits and tenant fair share. States a finite, documented retry contract with visible failure and replay. Refuses a false ordering promise and offers versions. Enforces SSRF defenses at connect time. Gives customers the delivery log so the on-call never debugs a customer's endpoint. Names who pays — failing customers absorb their own delays; healthy customers pay nothing. The interviewer should learn something from the answer.

10. Hot Takes#

10.1 "Guaranteed Delivery" Means "Guaranteed Within a Window — and Then We Tell You"#

The Staff position: Promise at-least-once within a stated window, then visible failure and replay. Anything stronger is untestable.

Why this matters in interviews: Accepting "guaranteed delivery" uncritically is a Senior signal; defining its window and end state is Staff.

10.2 Ordering Is Almost Always the Wrong Promise#

The Staff position: Put versions in the data and let receivers converge. Offer per-resource ordering only to customers who prove they need it.

Why this matters in interviews: Interviewers ask for ordering to see if you'll pay for it without asking the price.

10.3 The Delivery Log Is the Product#

The Staff position: A webhook platform without a customer-facing delivery log is half-built.

Why this matters in interviews: It shows you think about operational cost beyond your own pager.

10.4 Autoscaling Makes Slow Endpoints Worse#

The Staff position: Capacity is not the fix for a misbehaving tenant. Isolation is.

Why this matters in interviews: It's the most common wrong answer to the most common follow-up.

10.5 For Internal Consumers, Webhooks Are an Anti-Pattern#

The Staff position: If you control the receiver, let it pull from a log. Push-with-retries is a workaround for not controlling the other side; using it internally reinvents a broker badly, with HTTP timeouts instead of offsets.

Why this matters in interviews: Redirecting an internal-fanout framing to a broker in the first minute is a strong scoping signal.

11. Beyond Staff: The Principal View#

Why L7 Sees This Problem Differently#

The Staff engineer builds an isolated, secure, observable webhook platform. The Principal engineer notices that four product teams send webhooks independently — billing with its own retry loop, the repository service with no signing, the marketplace team with a different header format — and that customers integrating with two products see two contracts. The L7 problem is one event delivery contract for the company: a shared platform, a schema standard with resource versions, a published retry and signing contract, and a support model where customers self-serve.

The Org-Level Fault Line#

One delivery platform vs per-product senders.

🧭 Principal Move: "The platform owns delivery, signing, egress, retries and the delivery log. Products own event schemas, which must follow the standard — resource ID, monotonic version, no secrets in snapshots. Direct outbound HTTP to customer URLs from product services is blocked at the network after a two-quarter migration."

Cost Model#

Assumptions: async workers; columnar delivery log with compression; object storage for replay payloads at ~$0.02/GB-month; fully loaded engineer ~$250K/year.

The pricing insight: failure costs more than success. At growth scale, deliveries to failing endpoints and their log rows can be 40–60% of spend while being ~3% of endpoints. Disabling dead endpoints after 72 hours and storing excerpts only for non-2xx attempts are the two highest-ROI changes available.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

The Standard I'd Write#

RFC-EVENTS-002: Outbound Event Delivery Standard
Status: Approved   Owners: Event Platform + Security

Scope
  Every system that sends HTTP requests to customer-supplied URLs.

MUST
  1. Send only through the Event Delivery Platform; direct egress to customer
     URLs is blocked at the network.
  2. Include a stable event ID, resource ID and monotonic resource version.
  3. Exclude secrets and credentials from snapshot payloads; regulated data
     uses thin events only.
  4. Accept the platform's published delivery contract: at-least-once,
     unordered by default, documented retry window, signed requests.
  5. Register event types in the schema registry with backward-compatible evolution.

SHOULD
  1. Prefer thin events for payloads over 64KB.
  2. Coordinate scheduled bursts (billing runs) with the platform team.
  3. Provide a reconciliation API for every event type.

Exceptions
  Filed with Event Platform; security review required for any egress exception;
  time-boxed to two quarters.

Success metrics
  - Healthy-endpoint first-attempt p99: < 5s
  - Support tickets per million failed deliveries: tracked, trending down
  - Products sending webhooks outside the platform: 0 by end of year 2
  - SSRF blocks bypassed: 0

What I'd Tell the VP#

"Four teams send webhooks four different ways, and customers who integrate with two of our products get two contracts, two signature schemes and two support experiences. I'm proposing one event delivery platform: it costs about five engineers for a year, eliminates a class of security risk — servers making requests to arbitrary customer URLs — and lets customers debug their own integrations, which should cut webhook-related support tickets substantially. Product teams keep ownership of what their events say; the platform owns how they're delivered. The main risk is migration effort; I'd move the highest-volume product first."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll build a great webhook platform for billing" — correct, but ignores the three other senders.
• "Customers can see failed deliveries in the dashboard" — good, but no metric connects it to support cost.
• "We'll add a second region" — no mention of egress IP migration or residency for delivery logs.

Appendices

Appendix A: Mechanics in Depth#

A.1 The Scheduler Loop#

state:
  ready[endpoint]      : FIFO of delivery_ids (or per-resource heads if ordered)
  due_index            : min-heap of (next_attempt_at, delivery_id)
  inflight[endpoint]   : count, cap[endpoint]
  circuit[endpoint]    : CLOSED | OPEN(until) | HALF_OPEN
  deficit[tenant]      : weighted fair-share counter

every 10ms:
  move due retries from due_index into ready[endpoint]
  for tenant in round_robin(tenants_with_ready_work):
      deficit[tenant] += weight[tenant] * quantum
      for endpoint in tenant.endpoints_with_ready_work:
          while deficit[tenant] > 0 and inflight[endpoint] < cap[endpoint]
                and circuit[endpoint] != OPEN and ready[endpoint]:
              d = ready[endpoint].pop()
              lease(d, ttl=timeout+5s)          # persisted before dispatch
              dispatch(d); inflight[endpoint] += 1; deficit[tenant] -= 1

on_result(d, outcome):
  inflight[d.endpoint] -= 1
  record_attempt(d, outcome)                     # delivery log
  update_circuit(d.endpoint, outcome, latency)
  if outcome.success: mark_delivered(d)
  elif outcome.gone:  disable(d.endpoint); mark_failed_final(d)
  elif d.age > window: mark_failed_final(d)
  else: due_index.push(now + backoff(d.attempts) * jitter(0.8, 1.2), d)

A.2 Backoff Schedule#

attempt:   1    2    3     4    5     6     7    8    9..16
delay:     0    5s   30s   2m   10m   30m   1h   2h   every 3h until 72h
jitter:    ±20% on each delay; Retry-After honored up to 1h on 429/503

A.3 Circuit Breaker With Latency Awareness and Slow-Start#

window: last 60s, min 20 attempts
open if failure_rate > 50% or p99_latency > 10s
open → wait 30s (doubling to max 10m) → HALF_OPEN: send 1 probe
probe ok → CLOSED with cap = 2, double every 30s while success > 95%, up to configured cap
probe fail → OPEN with longer wait

Appendix B: Data Model#

CREATE TABLE endpoints (
  endpoint_id     TEXT PRIMARY KEY,
  tenant_id       TEXT NOT NULL,
  url             TEXT NOT NULL,
  event_types     TEXT[] NOT NULL,
  ordering        TEXT NOT NULL DEFAULT 'none',  -- none | per_resource
  max_concurrency INT NOT NULL DEFAULT 20,
  max_rate_per_s  INT,                           -- customer-set ceiling
  state           TEXT NOT NULL DEFAULT 'active',-- active | disabled
  disabled_reason TEXT,
  region          TEXT NOT NULL
);

CREATE TABLE endpoint_secrets (
  endpoint_id  TEXT NOT NULL REFERENCES endpoints,
  secret_id    TEXT NOT NULL,
  secret_enc   BYTEA NOT NULL,                   -- encrypted with KMS
  expires_at   TIMESTAMPTZ,                      -- NULL = active indefinitely
  PRIMARY KEY (endpoint_id, secret_id)
);

-- deliveries: hot state in the scheduler's store (sharded by endpoint),
-- persisted for durability; attempts go to the columnar delivery log:
-- attempts(tenant_id, endpoint_id, ts, delivery_id, event_id, attempt_no,
--          status_code, error_class, latency_ms, response_excerpt)  PARTITION BY day

Appendix C: Coordination Mechanisms#

C.1 Signing and Verification#

C.2 Quick Comparison#

Appendix D: API Contract & Client Behavior#

• Receivers should verify the signature, check the timestamp within 5 minutes, dedupe on the event ID, persist, return 2xx within 10 seconds, and process asynchronously.
• Receivers should treat events as unordered and use resource_version to ignore stale updates.
• 410 Gone disables the endpoint; any other non-2xx is retried per the schedule.
• Redirects are not followed; register the final URL.
• Secret rotation: POST /secrets:rotate with an overlap of up to 24 hours; both signatures are sent during the overlap.
• Replay: POST /endpoints/{id}/replay creates a job in a separate lane, rate-limited to the endpoint's configured max rate; replayed deliveries carry the original event ID and a Webhook-Replay: true header.

Appendix E: Observability#

Core metrics:

• delivery.first_attempt_p99{endpoint_health}, scheduler.oldest_ready_age{healthy}
• endpoint.inflight{endpoint}, endpoint.circuit_state, endpoint.circuit_flaps
• delivery.attempts_per_s{outcome}, events.failed_final_total{tenant}
• matcher.consumer_lag_seconds, ingest.lag_seconds
• egress.blocked_destination_total, egress.non_proxy_attempts
• dlog.ingest_lag_seconds, canary.receipt_gap_seconds

Critical alerts:

Debugging the silent failure: overall success rate and overall latency are useless — they're dominated by customer endpoints that are always failing somewhere. Watch healthy-endpoint latency, canary receipt and per-endpoint latency shifts (a sudden drop to single-digit milliseconds usually means something other than the customer's server is answering).

Appendix F: Scale Evolution#

What you don't build on day one: per-resource ordering, multi-region delivery, batched delivery, queue destinations, mTLS, a schema registry. Each has a trigger in Section 11.

Appendix G: Multi-Tenancy, Fairness & Cost#

• Fair share by tenant, weighted by plan, work-conserving; fairness by endpoint or by event is gameable.
• Per-endpoint caps by plan: free 5, standard 20, enterprise configurable with reserved capacity.
• Replay lane: replays never share the live lane's capacity; rate-limited to the endpoint's max rate.
• Cost attribution: attempts, retries and delivery-log bytes metered per tenant; heavy-failure tenants are visible in cost reports.
• Dead endpoint policy: disable after 72 hours of zero success; events remain replayable for 30 days — the single largest cost control.