Technologies referenced in this case study: Redis · DynamoDB · PostgreSQL · Apache Kafka · Envoy, Kong & NGINX · Kubernetes

Related: Edge Gateway · Rate Limiting · Distributed Cache · Multi-Region Active-Active · Graceful Degradation: Fail Open or Closed · Build vs Buy Framework · Replication

Reading Guide#

Organized for interview use first, reference second. This page covers issuing, storing, revoking and recovering identity. Verifying a token at the edge — JWKS caching, local signature checks, stripping client-supplied identity headers — lives in API Gateway; this page links there rather than repeating it.

Without revocation design:
t=0:        Malware on a user's laptop copies the browser's 90-day refresh token.
t=+2h:      Attacker replays it from another country. New access tokens issued.
t=+3h:      User notices odd purchases. Clicks "Log out of all devices".
t=+3h:      Logout deletes the user's sessions in one region's cache only.
t=+3h:      Attacker's 1-hour access JWT keeps working — nobody checks it against anything.
t=+4h:      Attacker refreshes again: refresh token is a stateless JWT too. Still valid.
t=+90 days: Refresh token finally expires. Support has 14 tickets. Trust gone.

With server-side sessions + rotation + revocation fan-out:
t=0:        Malware copies refresh token RT1 (family F9, bound to device key).
t=+2h:      Attacker presents RT1 without the device's proof-of-possession → rejected.
            (Weaker variant: no binding — attacker refreshes, gets RT2.)
t=+2h05m:   Real client presents RT1 (already rotated) → reuse detected → family F9 revoked.
t=+2h05m:   session.revoked event published. Gateways' deny list updated in ≤ 30s.
t=+2h06m:   Attacker's access token (10-min TTL) is denied by subject+session deny list.
t=+2h06m:   User forced to re-authenticate with a passkey. Security email sent.

Executive Summary

If you only read one section, read this. Everything in the case study flows from the contrast below.

What the Interviewer Is Scoring#

Authentication is not a crypto question. Nobody is asking you to implement ECDSA.

It is a revocation and blast-radius question that tests:

• Whether you can say how long a stolen credential stays useful, as a number, and what you'd do to shrink it
• Whether you know that every token you issue is a cache of a decision, and caches go stale
• Whether you design the identity service's failure — what still works when it is down — before its happy path
• Whether you treat account recovery as the real front door, because attackers do

The key insight: Every authentication design is a choice of where the "is this still valid?" check happens and how often. Check on every request and the identity store is in every request's critical path. Check never and a stolen token lives until it expires. Staff candidates pick the check frequency deliberately — per request, per refresh, per risk event — and state the revocation lag it buys.

One Question, Three Levels#

Positions to Commit To#

Which Problem Are We Solving?#

Three intents produce three different systems. Name them, then commit.

🎯 Staff Move: "I'll design consumer login — that's where the scale and the account-takeover pressure are. I'll keep the token and session model general enough that workforce SSO is a federation adapter on top, and I'll treat service-to-service identity as a separate platform problem that shares the signing-key discipline but not the session store. Tell me if you'd rather go deep on workforce or service identity."

Where the Design Splits#

How Real Companies Built It#

Why this section belongs here: Identity is the area where public incident reports teach the most. Each of these shows a fault line from this page playing out in production.

Facebook — "View As" and Access Tokens at Scale (2018)#

In September 2018 Facebook disclosed that three bugs interacting in its "View As" feature let attackers obtain access tokens — which Facebook describes as the digital keys that keep people logged in — for other users; it reset tokens for almost 50 million affected accounts plus another 40 million that had been subject to a "View As" lookup in the previous year, and turned the feature off. Its follow-up said about 30 million people actually had tokens stolen (Facebook security update, Facebook follow-up).

The mitigation was mass token reset — logging ~90 million accounts out — because that was the revocation primitive available at that scale. The bug was in a product feature, not the identity service, yet the identity system carried the response.

Staff insight: When asked "how would you respond to mass token theft?", the answer is a pre-built, rate-limited, targeted bulk revocation tool: revoke sessions by cohort (feature, time window, client version), not "everyone". Facebook reset an extra 40 million accounts as a precaution because the blast radius was hard to bound. Design so you can bound it.

Okta — Support-System Breach and Session Tokens in HAR Files (2023)#

Okta's root-cause report says that between 28 September and 17 October 2023 a threat actor accessed files in its customer support system belonging to 134 customers; some were HAR files containing session tokens, which were used to hijack sessions of five customers. Access came through a service account whose credentials had been saved to an employee's personal Google account, signed into a personal profile on an Okta-managed laptop. Remediations included disabling that account, blocking personal Google profiles on managed devices, and binding admin session tokens to network location (Okta root cause and remediation).

Staff insight: A session token is a bearer credential, and it will leak through channels nobody designed for — debug files, logs, support tickets. Two defenses follow: bind high-value sessions to something the attacker doesn't have (device key, network context), and keep a revocation path that support and security can trigger in seconds. In an interview, say "bearer tokens leak sideways; binding and fast revocation are how we make a leak boring."

Microsoft — Storm-0558 and a Consumer Signing Key (2023)#

Microsoft's investigation reported that Storm-0558 used an acquired Microsoft account (MSA) consumer signing key to forge tokens for Outlook Web Access and Outlook.com. Its leading hypothesis was that a 2021 crash of the consumer signing system produced a crash dump that, through a race condition, contained the key, and that the dump was later reached through a compromised engineering account; enterprise mail accepted the consumer-signed tokens because validation libraries did not automatically enforce key scope (Microsoft Security Response Center).

Staff insight: Two lessons for this page. First, signing keys are the crown jewels — keep them in HSMs or KMS and design emergency rotation as a rehearsed drill. Second, verification must check that a key is allowed to sign this kind of token for this audience, not only that the signature is mathematically valid. Say both in the key-management deep dive.

Google — BeyondCorp and Zanzibar#

Google's BeyondCorp paper describes moving access decisions off the network perimeter: corporate applications are reached over the internet, and access depends on the user and the device rather than on being inside a privileged intranet (Google research). Separately, Google's Zanzibar paper describes a central authorization system storing trillions of access control lists and serving millions of authorization requests per second at a 95th-percentile latency under 10 ms and availability above 99.999% over three years of production use (Google research).

Staff insight: These are the two ends of fault line 4. BeyondCorp says the network is not an identity; every request carries user and device identity. Zanzibar says a central authorization service can be fast and available enough — if you are willing to build what Google built. In an interview, cite Zanzibar to show central authorization is possible, then say what it costs before recommending it.

Follow-Ups to Expect#

System Architecture Overview#

Reading the diagram: Three paths with three very different rates. Verification (~1M/s) happens at the gateway with cached keys and never touches the identity plane. Refresh (~25K/s at peak) is the hottest identity endpoint and the place where "is this session still alive?" is answered. Login (~700/s legitimate, up to 50K/s during an attack) is the most CPU-expensive path and the one under attack. Revocation flows the other way — from the session store, through the event bus, out to every verifier — and its latency is the number this whole design is judged by.

One-Minute Recap#

Numbers to Bring#

Interview Walkthrough

The most common mistake: Candidates spend 20 minutes on the signup form, password hashing and the JWT payload, then run out of time before the interviewer asks the only question that matters: "A token was stolen. How long does it keep working, and what does it cost everyone else to stop it?" Compress the basics to ~10 minutes and spend the rest on revocation, the IdP's failure posture, credential stuffing and recovery.

Phase 1: Requirements & Framing (2–3 minutes)#

State the functional scope in one breath:

"Users sign up, log in with a password or a passkey, optionally add a second factor, stay logged in on several devices, log out of one or all devices, and recover their account when they lose credentials. Every other service in the company trusts the identity we issue."

Then the non-functional requirements, which is where the design lives:

"Four constraints drive everything. One: a stolen credential must stop working fast — I'll target 10 minutes worst case and 30 seconds for high-risk revocations. Two: identity is in every request's path, so an identity outage must not become a company outage — verification can't depend on the identity service being up. Three: login is under constant attack, so the expensive part — password hashing — must be protected from credential-stuffing volume. Four: recovery is a login path too, and usually the weakest one. Scale: I'll assume 200 million accounts, 60 million daily actives, ~700 logins per second at the morning peak, and around a million authenticated API requests per second."

Then name the underspecified parts:

"A few things I'd confirm: is this consumer, workforce or service-to-service? Do we have enterprise customers who need SSO into our product? Are there regulated accounts — payments, health — that need stronger assurance? I'll assume consumer, with enterprise SSO as a later adapter."

🎯 Staff Move: Putting a number on revocation lag in the first three minutes — "10 minutes worst case, 30 seconds for high-risk" — tells the interviewer you know the token model is a revocation decision, not a performance decision. Everything you draw next can be judged against that number.

Phase 2: Core Entities & API (1–2 minutes)#

Name the nouns in 30 seconds:

• Account: account_id, status (active, locked, recovering, deleted), assurance_tier, created_at
• Credential: credential_id, account_id, type (password, passkey, totp, phone, email, federated), secret_ref (hash or public key), added_at, trusted_after (cooling-off), last_used_at
• Session (refresh family): session_id, account_id, device_id, family_id, current_refresh_hash, prev_refresh_hash, rotated_at, auth_methods (pwd, passkey, otp), auth_time, expires_at, absolute_expires_at, revoked_at, binding_key_thumbprint
• SigningKey: kid, alg, status (pending, active, retiring, revoked), published_at, activated_at, retire_after
• AuditEvent: event_id, account_id, type, actor (user, support agent, system), ip, device, at

Client-facing API:

POST /v1/login                  { identifier, password | passkey_assertion, device_info }
  → 200 { access_token (10 min), refresh_token, expires_in }
  → 401 | 429 | 200 { step_up_required: ["otp","passkey"], challenge_id }

POST /v1/token/refresh          { refresh_token }   DPoP: <proof signed by device key>
  → 200 { access_token, refresh_token (new) }
  → 401 { error: "session_revoked" | "reuse_detected" }

POST /v1/sessions/{id}/revoke                         (log out one device)
POST /v1/sessions/revoke_all                          (log out everywhere, keeps current)
POST /v1/recovery/start         { identifier }
POST /v1/recovery/complete      { recovery_token, new_credential }
GET  /.well-known/jwks.json                           (public keys, cached by verifiers)

Login returns either tokens or a step-up challenge; the risk engine decides which. Refresh is the only endpoint that touches the session store on the hot path.

🎯 Staff Move: "I'm splitting the session from the token. The session is a row I can revoke. The access token is a 10-minute cache of 'this session was valid at issue time'. Once you see the token as a cache entry, its lifetime is just a TTL, and every cache-invalidation tool applies."

Phase 3: High-Level Architecture (≤5 minutes)#

Draw at most eight boxes:

Walk the login flow in 90 seconds:

1. Client submits credentials. The edge drops requests from known-bad IPs, ASNs and device fingerprints and applies per-IP and per-identifier rate limits — before any hashing.
2. Login service checks the identifier exists and isn't locked, then verifies the password with argon2id (~100ms) or a passkey assertion (~1ms signature verify).
3. Risk engine scores the attempt: new device, impossible travel, breached-password match, velocity. Outcome: allow, step-up, or block.
4. Token service creates a session row (refresh family), stores a hash of the refresh token, and signs a 10-minute access JWT with the active key in KMS.
5. Client calls APIs with the access JWT; the gateway verifies it locally with cached public keys — no identity call.
6. Every ~10 minutes the client refreshes: the token service looks up the session, checks it isn't revoked, rotates the refresh token, issues a new access token.
7. Logout or a risk event revokes the session row and publishes session.revoked; the distributor pushes the session ID to every verifier's deny list.

🎯 Staff Move: Say out loud: "Notice the three rates: verification at a million per second never touches identity; refresh at 25,000 per second is where revocation is enforced; login at 700 per second is the expensive one and the one under attack. I'm going to design each path to its own rate." You've now spent ~9 minutes.

Phase 4: Transition to Depth (1 minute)#

"That's the happy path, and it's the Senior-level design. What makes identity hard is what happens after a credential leaks and when the identity service fails. I'd like to go deep on four things: how fast we can revoke and how that reaches every verifier, what the rest of the company does when identity is down, how we survive a credential-stuffing wave without melting the password hashers, and how account recovery avoids becoming the takeover path. Where would you like to start?"

If no preference: start with revocation. It's the question that decides the level.

Phase 5: Deep Dives (25–30 minutes)#

For each: state the tradeoff → commit → quantify → name who pays.

Deep dive 1: Revocation (7–8 min)

"There are three revocation speeds and I'll be explicit about which events get which. Routine logout: revoke the session row — the access token dies at its natural expiry, at most 10 minutes later. High-risk events — password change, 'log out everywhere', takeover detected, admin kill: revoke the row and push the session ID to a deny list held in memory at every gateway node, with a 10-minute TTL, because after 10 minutes the token has expired anyway. Mass events — signing-key compromise: rotate the key, which invalidates every token signed with it."

Quantify: "High-risk revocations are maybe 0.5% of 60 million daily actives — 300,000 per day, 3.5 per second average. With a 10-minute TTL the deny list holds about 2,100 entries in steady state, under 100 KB. During a mass incident that could reach a few million entries — still under 200 MB, which a gateway can hold, but I'd switch to revoking by auth_time cutoff per account instead of listing session IDs."

Who pays: "The user pays up to 10 minutes of exposure for routine logouts. Security signed off on that; for anything they care about, the push path pays 30 seconds. The gateway team pays a small in-memory set and a subscription to the revocation stream."

Deep dive 2: The IdP is down (6–7 min)

"I'll split the dependency by operation. Verification: independent — cached JWKS. Refresh: depends on the session store; if the token service can't reach it, it issues a short extension — a 10-minute access token based on the still-valid refresh token's claims — for up to 30 minutes, but only if the client's refresh token is cryptographically valid and the session isn't on the local deny list. Login: fails closed. Step-up and sensitive actions: fail closed. That's a written policy security signs, not an on-call judgment call."

Quantify: "Without the grace, a 30-minute session-store outage expires every active access token within 10 minutes — 15 to 20 million concurrently active users logged out at once, then all of them retrying login when it comes back: 20 million logins in a few minutes against a 700/s design, or 30,000+ per second of argon2id. The grace prevents the outage from becoming a login stampede."

Deep dive 3: Credential stuffing (6–7 min)

"Stuffing is economics. Attackers try breached email/password pairs at volume. If every attempt reaches argon2id at 100ms, a 50,000/s wave needs 5,000 cores — the attack is a denial of service on login before it's a takeover. So the layers: IP and ASN reputation and per-IP rate limits at the edge; device and client-integrity signals; per-identifier limits — 10 failures per 15 minutes then step-up; a breached-credential check so a correct-but-breached password triggers a reset instead of a login; and the hash budget itself is a bulkhead: login gets a fixed pool, and when it's saturated we serve challenges, not timeouts."

Who pays: "Legitimate users behind shared IPs — mobile carriers, universities — pay extra challenges during waves. Product signs off on a challenge rate ceiling, say 2% of good logins."

Deep dive 4: Recovery (5–6 min)

"Recovery authenticates someone who has, by definition, lost their credentials, so it's the weakest proof we accept. Rules: notify every existing factor on start and completion; for accounts with a passkey or a high-value tier, a cooling-off of 24–72 hours before the new credential can change payout details or remove other factors; reset tokens are single-use, 15 minutes, bound to the browser that started the flow; and support-assisted recovery follows a scripted identity-proofing flow — agents can't skip steps or change contact details directly."

Phase 6: Wrap-Up (2–3 minutes)#

"The core idea: a token is a cache of an identity decision, so I picked the cache TTL — 10 minutes — and built a push path for the cases where 10 minutes is too long. Verification never depends on the identity service, so its outage degrades login, not the company. Login is protected economically, with shedding before the expensive hash, and recovery is treated as the front door it really is."

The evolution closer:

"What I'd build later: device-bound tokens via DPoP for all first-party clients; passkeys as the default credential, which removes most of the stuffing problem; cross-vendor revocation signals for enterprise customers through the OpenID shared-signals standards; and identity cells per region. What I'd not build: our own HSMs — cloud KMS is the right answer until a regulator says otherwise."

🎯 Staff Move: End on the revocation number and who owns it. Senior candidates end with "and we'd add MFA." Staff candidates end with "and revocation.propagation_p99 is an SLO the identity team carries, reviewed with security every month, because it's the number that decides how bad the next token theft is."

Common Timing Mistakes#

1. The Staff Lens#

1.1 Why This Problem Exists in Staff Interviews#

Identity is the dependency nobody can route around. A cache outage costs latency; a queue outage costs freshness; an identity outage costs every authenticated request in the company. At the same time it is the system attackers target most, because a single credential unlocks everything that identity guards. That combination — maximal blast radius on failure, maximal value on compromise — forces tradeoffs where both sides are expensive, and the candidate has to say who pays.

It also has an unusually deceptive happy path. A Senior engineer can build a login that works for every honest user on day one. The design is judged entirely by the dishonest ones: the credential stuffer, the malware that copies a refresh token, the caller who convinces a support agent they are the account owner. None of those show up in a demo. Staff candidates design for them before drawing the first box.

1.2 The L5 vs L6 Contrast — Visual#

1.3 The Staff Question That Cuts Through Everything#

"An attacker has a copy of one user's refresh token. Walk me through every place that token, and the access tokens derived from it, are accepted — and how long each keeps accepting it after the user hits 'log out everywhere'."

A candidate who answers with the refresh path (session store, revoked immediately), the access-token path (gateway, up to 10 minutes or ~30s via deny list), the downstream caches (services that cached the identity, bounded by the same TTL), the detection mechanism (rotation reuse, risk signals) and the owner (identity on-call for propagation, trust and safety for the account) has operated an identity system. A candidate who says "we delete the session" has built a login form.

2. Problem Framing & Intent#

2.1 The Three Intents — Explained#

Consumer login. The population is huge, mostly honest, and reuses passwords. The threat model is scale: attackers replay billions of leaked credential pairs, buy stolen session cookies, and socially engineer support. The design centers on cheap verification, aggressive abuse shedding, risk-based step-up, and recovery that is easy for real owners and hard for impostors. Product friction is a real cost here — every extra challenge loses a measurable fraction of logins — so security and growth negotiate every threshold. Correctness bar: account takeovers per million logins, revocation lag, login success rate.

Workforce SSO. The population is small (thousands to a few hundred thousand) but each account is worth far more: one administrator session can reach production, customer data or the identity system itself. The design centers on federation (OIDC and SAML to hundreds of SaaS apps), phishing-resistant MFA as a hard requirement, device posture, short sessions for privileged roles, and deprovisioning: a terminated employee must lose access everywhere within minutes, which means pushing revocation into apps you don't own. Correctness bar: time-to-deprovision, percentage of apps behind SSO, audit completeness.

Service-to-service identity. No humans, no passwords, no recovery flow. Millions of calls per second between workloads that are created and destroyed constantly. The design centers on the platform attesting what a workload is (which cluster, namespace, service account) and issuing short-lived credentials — X.509 certificates for mTLS or signed tokens — that rotate automatically. Standards like SPIFFE define the identity format and how workloads fetch short-lived identity documents (SPIFFE overview). Correctness bar: zero long-lived secrets, rotation success rate, and no deploy blocked by the identity platform. This intent pairs with Service Registry and the workload model in Kubernetes.

🎯 Staff Move: "These share almost nothing operationally. Consumer is about abuse at scale, workforce is about high-value accounts and deprovisioning, service identity is about automation and rotation. If someone asks me to build one system for all three, I'd share the key-management and audit layers and nothing else."

2.2 When NOT to Build Your Own Identity Provider#

And within the design, some features you should not build even when you own the IdP:

• Don't invent a token format. Use JWT/JWS or opaque tokens per the OAuth and OIDC specs. Custom formats lose the libraries, the audits and the interoperability.
• Don't build the resource owner password credentials grant. The OAuth 2.0 Security Best Current Practice says it MUST NOT be used (RFC 9700).
• Don't store long-lived secrets for services when the platform can attest them.
• Don't run your own HSM fleet unless a regulator requires it; cloud KMS gives non-exportable keys with audit logs.

The Staff signal is knowing that identity is the textbook case where buying is the default and building needs a reason — scale economics past ~50–100M MAU, unusual assurance or data-residency requirements, or identity being the product itself. See Buy or Build: The Total-Cost Test and fault line 5.

2.3 What the Interviewer Leaves Underspecified#

2.4 Precise Terminology#

3. Where the Design Splits#

Each fault line below follows the same shape: the options, who pays for each, the Staff default, and when to deviate.

3.1 Fault Line 1: Stateless Tokens vs Server-Side Sessions#

The tension: A server-side session is revocable the instant you delete it, but every request needs a lookup. A signed token needs no lookup, but once issued it is valid until it expires, wherever it has been copied.

The Staff default: hybrid plus deny list. 10-minute access JWTs verified locally (the mechanics are in Edge Gateway), opaque refresh tokens checked against a server-side session store at refresh time, and a pushed deny list for high-risk revocations. The arithmetic: at 1M API requests/s and one refresh per active client per 10 minutes, the session store sees ~25K reads/s instead of 1M — a 40× reduction — and revocation lag is 10 minutes worst case, ~30 seconds for events that matter.

"I want the session store in the refresh path, not the request path. That's a 40× cut in dependency load, and I buy back revocation speed with a push channel for the 0.5% of revocations where minutes matter."

When to deviate:

• Low-volume, high-assurance apps (admin consoles, banking back-office): opaque sessions with per-request lookup. 200 requests/s doesn't need a token cache, and instant revocation is worth 2ms.
• Server-rendered web apps on one backend: a classic cookie session in Redis is simpler and fine; JWTs add nothing when the verifier and the session store sit together.
• Offline-capable clients: longer access tokens are sometimes unavoidable; compensate with device binding.

3.2 Fault Line 2: Token Lifetime — Short + Refresh vs Long-Lived#

The tension: Every minute of access-token lifetime is a minute a stolen copy works. Every minute you remove is refresh load and a chance for a flaky mobile network to log someone out.

Refresh-token lifetime is a separate decision: how long a device stays logged in without re-authenticating. Consumer default here: 30-day sliding, 90-day absolute, with step-up for sensitive actions based on auth_time, not on whether a session exists.

Rotation and reuse detection. Every refresh returns a new refresh token and invalidates the old one. If an old token is presented again, either the client retried or someone else has a copy — the server can't tell which, so it revokes the whole family. The OAuth security BCP requires refresh tokens for public clients to be either sender-constrained or rotated (RFC 9700).

The race: a mobile client sends RT1, the server rotates to RT2, the response is lost, the client retries with RT1. Without a grace window, that's a false theft signal and a logged-out user. The fix: accept the immediately previous token for 30–60 seconds from the same device binding and return the same RT2 (idempotent rotation). Outside that window, reuse revokes the family.

Who pays: short access TTLs are paid by the identity team (refresh capacity) and mobile users (battery, a few hundred bytes per refresh). Long TTLs are paid by the users whose tokens are stolen. Rotation is paid by users on bad networks if the grace window is wrong — watch refresh.reuse_detected_total split by client version; a spike after a mobile release is a client bug, not an attack.

The Staff default: 10-minute access, rotated refresh with a 60-second idempotent grace, family revocation on reuse, DPoP binding for first-party clients when the client platform supports secure key storage (RFC 9449).

3.3 Fault Line 3: Fail Closed vs Fail Open When the IdP Is Unreachable#

The tension: If the identity plane is down and you fail closed, nobody can do anything — the identity outage becomes a company outage. If you fail open, you might accept a session that was revoked during the outage. See the general framework in Graceful Degradation: Fail Open or Closed.

The mistake is answering this once for the whole system. The Staff answer splits it by operation:

The grace extension, precisely: the token service (or a regional fallback signer) can verify the refresh token's own signature or MAC without the session store if refresh tokens are self-verifiable — for example an opaque random value plus an HMAC over session_id and expiry. It issues a 10-minute access token marked grace=true, repeats at most 3 times (30 minutes), and never for sessions on the local deny list. Services that care (payments, admin) reject grace=true tokens for sensitive operations.

Who signs off: security owns the maximum grace duration; product owns which features accept grace=true. It's a signed table, reviewed twice a year, not a runbook improvisation at 3 a.m.

🎯 Staff Move: "I'd rather let a user whose session was revoked during a 20-minute outage keep browsing for 20 minutes than log 20 million people out and then take 20 million logins in five minutes when we come back. But password changes and payouts fail closed, and security signs the table that says so."

3.4 Fault Line 4: Central Authorization Service vs Policy in Each Service#

The tension: Authentication answers "who". Authorization answers "may they do this to that". A central authorization service gives one consistent answer and one audit trail, and becomes a dependency of every request. Policy inside each service is fast and independent, and drifts.

The Staff default: keep authentication central and authorization close to the data. Tokens carry identity and a small, stable set of coarse claims (tenant, account tier, a few scopes) — not permissions. The gateway enforces coarse checks; services enforce resource-level decisions with a shared policy library and a common decision-log format. Move to a central relationship service only when the product's permission model is genuinely a graph (document sharing, nested orgs) and several services need the same answers — that's what Zanzibar was built for, at a cost most companies shouldn't pay first.

When to deviate: collaborative products with sharing (docs, drives, design tools) hit the graph problem early; for them a central relationship service is the right Year-1 investment. See Collaborative Documents for the product side.

3.5 Fault Line 5: Build vs Buy the Identity Provider#

The tension: Identity is the most security-critical code you'll run, which argues for buying it from a specialist. It's also in every request and every product decision about signup friction, which argues for owning it. See Build vs Buy Framework.

The Staff default: buy workforce identity, always. For consumer identity, buy below ~10M MAU, decide on the numbers between 10M and 100M, and build or self-host above ~100M MAU or when login UX and risk logic are core to the product. Whatever you choose, own the session and revocation layer's interface: products should depend on your token contract, not the vendor's SDK, so a vendor change is a migration, not a rewrite.

The vendor-incident caveat. Buying moves the work, not the risk. The Okta support-system incident above shows a vendor breach reaching customer sessions. If you buy, you still need: your own revocation path, your own monitoring of admin sessions, and a contract that says how fast the vendor notifies you.

4. When It Breaks#

4.1 Signing-Key Leak — The Emergency Rotation#

t=0:       Security finds the active token-signing private key (kid=k42) in a debug
           artifact uploaded to a third-party ticketing tool 6 hours ago.
t=+5min:   Incident declared. Assume forged tokens for any user, any audience.
t=+10min:  Generate k43 in KMS (non-exportable). Publish k43 in JWKS alongside k42.
t=+12min:  Mark k42 "revoked" in the verifier config channel — pushed, not waiting
           for JWKS TTL. Gateways reject kid=k42 within 60s.
t=+13min:  Every access token signed by k42 now fails: ~18M active clients get 401.
t=+13min:  Clients refresh. Refresh tokens are opaque and checked against the session
           store — unaffected by the JWT key — so refresh succeeds with k43 tokens.
t=+15min:  Refresh traffic spikes from 25K/s to ~180K/s. Token service autoscaled from
           pre-warmed capacity, plus jittered client retry (0–60s).
t=+25min:  Refresh traffic back under 40K/s. 0.4% of clients failed and fell back to login.
t=+2h:     Audit: tokens signed with k42 after the leak window carrying unusual audiences
           or sessions that don't exist in the session store = forged. 0 found.

Why it went this well: refresh tokens were not signed by the same key as access tokens, so a JWT-key leak didn't force a password re-login for 60M users. Verifiers honored a pushed key-revocation signal instead of waiting for JWKS cache expiry. Refresh capacity was sized for a "everyone refreshes at once" event.

Detection: tokens.verified_with_unknown_session_total (a valid signature on a session that doesn't exist is a forgery signal), secrets scanning on outbound artifacts, KMS Sign call volume by caller.

Prevention: keys in KMS or HSM, never exportable; signing only through a narrow signer service; per-audience or per-token-type keys so one leak doesn't forge everything; verifiers enforce iss, aud and key scope, not only signature validity — the lesson Microsoft published after Storm-0558.

Owner: identity team (rotation); security (incident command); gateway platform (verifier key config).

4.2 IdP Outage Locks Every Service#

t=0:       Session-store primary in us-east fails over; replica promotion stalls
           (replication lag 40s, failover guard waits). Refresh calls time out.
t=+1min:   refresh.error_rate 0.1% → 92%. Access tokens keep working (local verify).
t=+10min:  Without grace: first wave of 10-min access tokens expires. Users see 401s.
           Apps call refresh in a tight loop: 25K/s → 400K/s retries.
t=+12min:  Token service pods CPU-pinned by retries. Login also shares the pool.
t=+15min:  "Can't log in" trends on social media. Every product reports outage.
t=+25min:  Session store recovers. 15M clients all fall back to full login.
t=+26min:  Login at 40K/s → argon2id pool saturated → login p99 30s → more retries.
t=+70min:  Stable after login admission control and client backoff hotfix.

With the Staff design:
t=+1min:   Refresh fails → token service issues grace tokens (grace=true, 10 min, max 3).
t=+10min:  Users notice nothing. Payouts and password changes show "temporarily unavailable".
t=+25min:  Session store back. Grace tokens refresh normally. No login stampede.

Detection: refresh.error_rate, refresh.grace_issued_total, session_store.replication_lag_seconds, login.queue_depth.

Mitigation: grace extension (fault line 3); separate thread pools for refresh and login so a refresh storm can't starve login; clients with exponential backoff and jitter on refresh, enforced in the shared client SDK; Retry-After honored.

Prevention: session store replicated within the region with automatic failover tested monthly; regional cells so one region's session store never affects another (Multi-Region Active-Active); quarterly IdP-down game day.

Owner: identity on-call (primary); client platform team owns SDK backoff behavior.

4.3 Credential-Stuffing Wave#

t=0:       07:40 local, morning ramp: 600 legit logins/s.
t=+1min:   Attack starts: 45K login attempts/s from ~220K residential proxy IPs,
           each IP sending < 1 attempt/min. Per-IP limits don't trigger.
t=+2min:   argon2id pool (2,000 cores, ~20K hashes/s) saturated. Legit login p99 → 12s.
t=+3min:   Page: login.attempts_rate 75× baseline, login.success_rate 94% → 3%.
t=+5min:   On-call enables wave mode: unknown-device attempts get a challenge before
           hashing; identifiers seen in breach corpus with no prior device → step-up.
t=+8min:   Hash pool load drops to 30%. Legit success rate back to 88% (challenge friction).
t=+30min:  ~0.6% of attempted pairs were valid. Those accounts: forced reset + notification,
           sessions created during the wave revoked.
t=+2h:     Attack stops. Challenge rate decays back to 0.3%.

Detection: login.attempts_rate vs baseline, login.success_rate (a sharp drop is the stuffing signature — most pairs are wrong), login.unique_identifiers_per_min, ratio of failures on non-existent identifiers.

Mitigation: layered shedding before the hash: client-integrity and device signals, ASN and proxy reputation, per-identifier limits, challenges for unknown devices; a fixed hash budget as a bulkhead so overload produces challenges, not timeouts.

Prevention: breached-password screening at signup and at login (NIST SP 800-63B requires checking new passwords against a blocklist of compromised ones); passkey adoption, which removes the password entirely; notify users on new-device logins.

Owner: identity on-call for capacity; trust and safety for the account remediation; product signs off on the challenge-rate ceiling. Rate-limiter mechanics: Rate Limiter.

4.4 Token Cache Serving Revoked Sessions#

t=0:       A team adds a 1-hour in-process cache of "token → user profile + session valid"
           to cut calls to a profile service. Nobody reviews it as an auth change.
t=+3 weeks: User reports takeover. Support runs "log out everywhere" at 14:02.
t=+3 weeks: Gateway deny list blocks the session at 14:02:20. But the internal service
           behind a second ingress path (partner API) uses its own cache: still serves
           the attacker until 15:02.
t=+3 weeks: Attacker exports 1,200 contacts in that hour.

Detection: this is a silent failure — nothing errors. Catch it with a synthetic canary: every 5 minutes, create a session, revoke it, then probe every ingress path with its access token; alert on any 2xx after revocation_time + 60s. Metric: revocation.canary_accept_after_revoke_total.

Mitigation: purge the cache; subscribe that service to session.revoked.

Prevention: a standard: any cache of identity or session validity MUST have TTL ≤ access-token lifetime and MUST subscribe to revocation events; the canary covers every ingress, not just the main gateway. Generic cache-invalidation tradeoffs: Distributed Cache.

Owner: identity team owns the canary and the standard; the service team owns the fix.

4.5 Account Recovery as the Takeover Path#

t=0:       Attacker controls victim's email via a separate breach.
t=+2min:   Requests password reset. Link arrives in the victim's (now attacker's) inbox.
t=+5min:   Old design: password changed, all sessions revoked — including the real owner's.
           Attacker removes the passkey and adds their own phone as 2FA.
t=+1 day:  Owner locked out of their own account. Support can't verify them because
           every factor on file now belongs to the attacker.

With cooling-off:
t=+5min:   New password works, but account enters limited mode for 24–72h. The existing
           passkey device receives "Someone is recovering your account — this wasn't me".
t=+40min:  Owner taps "this wasn't me". Recovery cancelled, attacker sessions revoked,
           email flagged as compromised.

Detection: recovery.started_rate, recovery.cancelled_by_owner_rate, recovery.success_then_dispute_rate (recoveries later reported as takeovers), support-assisted recoveries per agent per day.

Mitigation: freeze the account, revoke all sessions, restore from the credential history in the audit log.

Prevention: cooling-off and limited mode for high-value tiers; never let recovery remove existing factors immediately; support tooling that can't change contact details without the scripted flow; encourage two independent recovery factors.

Owner: recovery policy — security sets the floor, product sets friction above it; recovery system — identity team; support-assisted recovery — support operations, audited monthly by security.

4.6 Key Rotation Timeline (Routine)#

Routine rotation is the drill that makes 4.1 survivable. The verifier side — caching JWKS and refreshing on an unknown kid — is covered in Edge Gateway; the issuer side is here.

Cadence: every 30 days, fully automated, with jwks.verifiers_missing_active_kid (verifiers that fetched JWKS but don't have the active key) as the pre-flight check. An emergency rotation is the same pipeline with the waits removed and a push signal added.

4.7 Operational Reality Matrix#

🎯 Staff Insight: The most dangerous identity failure is the one that doesn't error: a revoked session that keeps working. Every other row here pages. That one needs a synthetic canary that revokes a session every 5 minutes and checks every ingress path — otherwise you discover it from a user's takeover report.

5. Scorecard#

5.1 Level-Based Signals#

5.2 Strong Hire Signals#

5.3 Lean No-Hire Signals#

5.4 Common False Positives#

• OAuth flow fluency ≠ identity design. Reciting the authorization code flow with PKCE is table stakes; the Staff question is what happens after the token is issued.
• Crypto depth ≠ security judgment. Comparing RSA-2048 with Ed25519 impresses briefly; revocation lag and key management decide whether a breach is contained.
• "Zero trust" vocabulary ≠ a design. It's Staff only if the candidate names what each request carries (user identity, device identity) and where it's verified.
• MFA everywhere ≠ safety. SMS OTP is phishable and SIM-swappable; a candidate who stops at "add MFA" hasn't thought about phishing-resistant factors or recovery bypassing them.

6. The 45 Minutes, Phase by Phase#

6.1 Typical 45-Minute Shape#

6.2 How Interviewers Pivot — And What They're Testing#

6.3 What to Deliberately Skip#

• Password complexity rules. One sentence: "length minimum, breached-password check, no composition rules, per NIST 800-63B."
• Every OAuth redirect. Say "authorization code with PKCE" and move on unless asked.
• Email-verification UX. Mention it exists.
• Crypto algorithm selection. "ES256 or EdDSA, keys in KMS."
• CAPTCHA vendor details. "Challenge" is enough; the decision is when to challenge.

6.4 Follow-Up Questions to Expect#

1. "A user clicks 'log out everywhere'. List every place a stolen token is still accepted, and for how long."
2. "The session store is down for 20 minutes. What happens to logged-in users, and to new logins?"
3. "A mobile release causes 2% of users to be logged out every day. What's your first hypothesis?"
4. "How do you migrate 200 million legacy SHA-1 password hashes to argon2id?"
5. "An enterprise customer fires an employee. How fast does that person lose access to our product?"
6. "How do you rotate signing keys without breaking anyone, and how do you do it in an emergency?"
7. "Who decides how long account recovery takes for a high-value account?"

7. Practice Rounds#

Drill 1: The Opening#

Prompt: "Design authentication for our app."

Drill 2: Revocation Mechanics#

Prompt: "A user reports their account was compromised and clicks 'log out of all devices'. Walk me through exactly what happens."

Drill 3: Make It Concrete — Capacity#

Prompt: "Size the identity plane. How many machines, how much storage?"

Drill 4: The Identity Provider Is Down#

Prompt: "Your session store is unreachable in one region for 25 minutes. What does every other team experience?"

Drill 5: Credential Stuffing at the Morning Peak#

Prompt: "45,000 login attempts per second just started, from 200,000 IPs. Most fail. What do you do?"

Drill 6: Multi-Tenant Enterprise SSO#

Prompt: "B2B customers want their employees to log in with their own IdP. One customer has 200,000 employees."

Drill 7: Build vs Buy#

Prompt: "Should we build our own identity provider or buy one?"

Drill 8: Changing Token Lifetime Without an Outage#

Prompt: "Security wants access tokens cut from 60 minutes to 10. How do you roll that out?"

Drill 9: Multi-Region#

Prompt: "We're expanding to the EU and must keep EU users' identity data there."

Drill 10: Cost#

Prompt: "Identity costs are growing faster than users. Where's the money going?"

8. Incident Walkthroughs#

Deep Dive 1: Peak-Traffic Incident — Launch-Day Login Meltdown#

Context: A new product launches with a TV campaign. Signups and logins hit 12× normal in 10 minutes; at the same moment a stuffing botnet piggybacks on the traffic. Login p99 is 25 seconds and the CEO's demo account can't sign in. The on-call escalates to you.

Questions to Surface First:

• How much of the traffic is legitimate? What's login.success_rate doing — dropping (stuffing) or steady (real users)?
• Is the hash pool saturated, or is something upstream (risk engine, credential store) the bottleneck?
• Are signups and logins sharing the same pool?
• Did the client release for the launch change refresh or retry behavior?

Typical L5 Approach: Scales up the login service and adds per-IP rate limits. The scale-up takes 8 minutes, the botnet's per-IP rate is already below the limit, and new pods saturate on the same hashing cost.

Staff Approach: Separates signal from noise first: success rate fell from 94% to 21%, so most traffic is stuffing. Enables wave mode (challenge unknown devices before hashing), splits signup into its own pool so launch signups aren't starved, and finds the launch client retried failed logins 3 times with no backoff — tripling legitimate load.

Principal Approach: Treats launch-day identity readiness as a launch-checklist item owned by identity: capacity sign-off, wave-mode rehearsal, client SDK version gate. Makes "identity load test at 20× with attack mix" mandatory for any campaign with paid media.

Key Takeaway: "Login capacity is sized for the attack, and the shedding decision is pre-approved — not debated during the incident."

What clears the Staff bar:

• Reads success rate before scaling
• Protects the hash pool with shedding, not more pods
• Finds the client-retry multiplier

Deep Dive 2: Silent Failure — Logouts Haven't Propagated for Three Days#

Context: A security researcher reports that after "log out everywhere", their old access token still worked for exactly 10 minutes — not 30 seconds. Investigation shows the deny-list distributor stopped consuming three days ago after a schema change. Nothing paged.

Questions to Surface First:

• Did revocations still work at the refresh path? (Yes — session store was fine, so lag fell back to 10 minutes, not infinity.)
• Which high-risk revocations happened in those three days, and what did those sessions do in the 10-minute windows?
• Why didn't consumer lag or the canary alert?

Typical L5 Approach: Restarts the distributor, fixes the schema mismatch, closes the bug.

Staff Approach: Fixes the consumer, then fixes detection: consumer lag was monitored but routed to a dashboard, and the revocation canary only probed the refresh path, not the gateway. Adds the canary on every ingress with a 60-second threshold, and reviews the 4,100 high-risk revocations from the window for activity after revocation.

Principal Approach: Recognizes that the system degraded gracefully by design — the 10-minute TTL bounded the damage — and that the gap is in owning the SLO. Makes revocation.propagation_p99 a paging SLO, adds schema-compatibility checks to the event bus for every security-relevant topic, and reports the incident in the security KPI review.

Key Takeaway: "Bounded failure is the reason the 10-minute TTL exists. Measuring the fast path is the reason the canary exists."

What clears the Staff bar:

• Notices the access TTL bounded the damage, and says so
• Measures the end-to-end revocation result, not component health
• Quantifies exposure from the incident window

Deep Dive 3: Large-Customer Onboarding — A 200,000-Employee Bank#

Context: Sales signs a bank with 200,000 employees. Requirements: SAML SSO through their IdP, phishing-resistant MFA, sessions ≤ 8 hours, deprovisioning within 15 minutes, every admin action logged and exportable, and an audit of our identity controls before go-live.

Questions to Surface First:

• Does their IdP support SCIM and push signals, or do we poll?
• Do they want their MFA policy or ours to govern? (Theirs — we record their amr.)
• Where must their audit logs live, and for how long?

Typical L5 Approach: Adds a SAML connection and a tenant flag for 8-hour sessions.

Staff Approach: Treats deprovisioning as the hard requirement: SCIM deactivate → revoke all sessions for those users → push deny entries; plus a 15-minute access-token cap for this tenant and a nightly SCIM full-sync to catch missed events. Enforces tenant policy (session length, IP ranges) in the token service, not in the app. Audit events per tenant, exportable via API.

Principal Approach: Turns the bank's requirements into a standard enterprise tier — session policy, deprovisioning SLA, audit export — priced as a product SKU, and accepts continuous-access signals from customer IdPs so their "user disabled" reaches our sessions in seconds without polling.

Key Takeaway: "For enterprises, the login is easy. Deprovisioning is the product."

What clears the Staff bar:

• Puts deprovisioning latency at the center with a measured SLA
• Keeps tenant policy enforcement in the token service
• Isolates the big tenant's sync load

Deep Dive 4: Post-Mortem — High-Value Accounts Taken Over Through Support#

Context: Over two weeks, 23 high-follower accounts were taken over. None of the attackers knew a password. All 23 went through support-assisted recovery. You own the post-mortem.

Questions to Surface First:

• What did the attackers present to support, and which agent actions moved contact details?
• Did existing factors get notified, and were notifications delivered to the attacker's new contact details?
• Is there an insider or a single agent pattern?

Typical L5 Approach: Retrains support agents, adds a security question to the support script.

Staff Approach: Finds the control gap: agents could change the email address directly, and the recovery flow then sent everything to the new address. Removes direct contact edits from support tooling, routes all support recovery through the same cooling-off flow with notifications to the previous factors, and adds second-agent approval for high-value tiers.

Principal Approach: Treats support as part of the authentication system. Assigns recovery-policy ownership to security with product and support as consulted parties, audits support-assisted recoveries monthly, and measures recovery.success_then_dispute_rate per channel and per agent as a standing security metric.

Key Takeaway: "Every human who can change a credential is part of the authentication system."

What clears the Staff bar:

• Finds the tooling permission, not just the training gap
• Notifies previous factors, which the attacker doesn't control
• Measures recovery abuse per channel

Deep Dive 5: Multi-Region Expansion — Identity Cells#

Context: The company runs identity in one US region. A regional cloud outage took login down globally for 3 hours last quarter. Leadership wants regional independence and EU residency within a year.

Questions to Surface First:

• Which identity data must be homed (credentials, sessions, audit) and which can be global (public keys, routing directory)?
• How will a user whose home region is down be treated?
• Which product services call the identity plane synchronously today?

Typical L5 Approach: Deploys a second active region with a globally replicated user database.

Staff Approach: Cells per region with home-region accounts. Global, read-only directory mapping hashed identifier → home region. Public keys published globally so verification works anywhere. Revocation events replicated to all regions. A home-region outage affects only that region's users and grace mode covers 30 minutes for them.

Principal Approach: Sequences the migration as a multi-quarter program with an explicit order — verification independence first, then refresh, then login — and makes "no synchronous cross-region identity call" a design-review rule for every product service.

Key Takeaway: "Identity becomes regional when its outage stops being global."

What clears the Staff bar:

• Distinguishes homed from global identity data
• Keeps the one global component off the critical write path
• Plans migration in batches with dual-read

9. Level Expectations Summary#

After studying this case study, you should be able to:

• State revocation lag as a number for any token design and explain how to shrink it
• Design the hybrid model: short access JWT, server-side refresh sessions with rotation and reuse detection, and a pushed deny list
• Size login, refresh and verification as three separate paths with their own rates
• Split IdP failure posture by operation and defend the grace extension with a sign-off
• Explain credential stuffing as a CPU and economics problem and shed it before the hash
• Design account recovery with cooling-off, notifications and support controls
• Run a routine and an emergency signing-key rotation, issuer side and verifier side
• Decide when to buy an identity provider and how to keep the decision reversible

The Bar for This Question#

Mid-level (L4): Builds a working login: hashed passwords, a session cookie or a JWT, a password reset email, maybe TOTP. Doesn't consider revocation lag, outages or abuse. Would ship something that works for honest users and fails the first takeover report.

Senior (L5): Adds MFA, rate limiting, refresh tokens and replicas. Knows JWTs are hard to revoke. The gap: treats revocation as a database delete, sets token lifetimes without connecting them to exposure, answers "IdP down" with "replicas", and sends a reset link to an email that may already be compromised. The design is plausible and would pass review — and would still turn a stolen refresh token into a 90-day problem.

Staff+ (L6): Frames the problem around revocation and blast radius in the first five minutes. Chooses the hybrid token model with numbers — refresh load, deny-list size, lag. Splits failure posture by operation, sheds stuffing before hashing, treats recovery as a login path, and runs a canary for the silent failure. Names who pays each tradeoff: users carry bounded exposure, product carries challenge friction, security signs the grace table. The interviewer should learn something from the answer.

10. Hot Takes#

10.1 "Stateless JWT" Is a Revocation Decision Disguised as a Performance Decision#

The Staff position: Every token design has state somewhere. Choose where — per request, per refresh, per risk event — and state the lag it buys.

Why this matters in interviews: "Stateless" is the word that makes interviewers ask about logout. Have the number ready.

10.2 Account Lockout Is a Gift to Attackers#

The Staff position: Throttle and challenge; don't lock. NIST's 100-attempt ceiling is a ceiling, not a design.

Why this matters in interviews: "Lock the account" is the most common stuffing answer and the easiest to dismantle.

10.3 Recovery Is the Real Login Page#

The Staff position: Whatever your strongest factor is, your account is only as strong as the weakest way to replace it. A passkey-protected account with email-only recovery is an email-protected account. Design recovery first, then login.

Why this matters in interviews: Bringing up recovery unprompted is one of the clearest Staff signals on this question.

10.4 Most Companies Should Not Build Central Authorization#

The Staff position: Zanzibar proves central authorization can be fast. It doesn't prove you should build it. Wait for the graph.

Why this matters in interviews: Candidates who propose a central authorization service at minute 10 have added a tier-0 dependency without a reason.

10.5 SMS Is a Cost Center and a Liability#

The Staff position: SMS OTP is phishable, SIM-swappable, and an attack target for pumping fraud that bills you per message. Keep it as a fallback, make passkeys and TOTP the default, and watch SMS spend by destination country like a security metric.

Why this matters in interviews: Naming SMS's cost and abuse profile shows you've seen the bill, not just the design doc.

11. Beyond Staff: The Principal View#

Why L7 Sees This Problem Differently#

The Staff engineer designs a correct identity system. The Principal engineer notices that the company has five — the original consumer login, the acquired product's login with its own user table, an enterprise SSO bolted on by the B2B team, a partner API using static keys, and service-to-service calls authenticated by a shared secret from 2019. Each has its own token format, its own lifetime and its own idea of "log out". The L7 problem is not one correct login; it is identity as a company-wide trust boundary: one revocation story, one key-management discipline, one recovery policy with a named owner, and a migration path that retires the other four without a flag day.

🧭 Principal Move: "I'm not going to ask whether our login is secure. I'm going to ask what the longest-lived credential in circulation is, which system issued it, and who would notice if it were stolen. That one number tells me where to spend the next year."

The Org-Level Fault Line#

One identity platform vs per-product login.

🧭 Principal Move: "The platform owns everything that must be right once — credential storage, sessions, token issuance, keys, revocation and the recovery floor. Products own everything that must be fast to change — onboarding screens, copy, which step-up method to offer above the floor. New products may not create a user table; the two existing ones migrate within four quarters."

Cost Model#

Assumptions: consumer-heavy, fully loaded engineer ~$250K/year, cloud list prices, SMS at ~$0.03/message blended, vendor CIAM priced per MAU at negotiated rates that fall with volume (estimates, not quotes).

The pricing insight: at growth scale, the two levers worth more than any infrastructure tuning are SMS volume and attack headroom. Moving 30% of step-ups from SMS to passkeys or TOTP saves ~$10–20K/month and removes a phishing vector; shedding stuffing before the hash avoids sizing the hash pool for the attack. Headcount, not infrastructure, is the dominant cost at every scale.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

🧭 Principal Insight: The passkey relying-party ID and the account identifier model are the decisions I'd slow down on. Everything about tokens can change in a quarter; those can't change in a year.

The Standard I'd Write#

RFC-ID-001: Authentication and Session Standard
Status: Approved   Owners: Identity Platform + Security Engineering

Scope
  Every system that authenticates users, issues or accepts user or service
  credentials, or caches identity or session validity.

MUST
  1. Authenticate users only through the Identity Platform API or hosted login.
  2. Access tokens: lifetime ≤ 15 min; refresh tokens: server-side state,
     rotated, reuse revokes the family.
  3. Any cache of identity or session validity: TTL ≤ access-token lifetime and
     subscribed to session.revoked.
  4. Every ingress path passes the revocation canary before launch.
  5. Signing keys live in KMS/HSM, are non-exportable, rotate ≤ 30 days, and are
     scoped per token type and audience. Verifiers check iss, aud and key scope.
  6. Service-to-service authentication uses platform-issued workload identity;
     no static shared secrets.
  7. Account recovery follows the tiered recovery policy owned by Security.

SHOULD
  1. Offer passkeys as the default credential for new accounts.
  2. Sender-constrain first-party tokens (DPoP or mTLS).
  3. Reject grace-mode tokens for sensitive operations.

Exceptions
  Filed with Identity Platform; reviewed within 10 business days; time-boxed to
  2 quarters; CISO sign-off for any exception to MUST 2, 5 or 7.

Success metrics
  - revocation.propagation_p99 ≤ 30s; canary failures: 0
  - Longest-lived user credential in circulation ≤ 90 days
  - ATO per million logins: tracked monthly, target set yearly by Security
  - Static service secrets in config: 0 by end of year 2
  - Identity-caused product downtime: ≤ 20 min/quarter

What I'd Tell the VP#

"We have five ways to log in and no single way to log someone out. If one of our users' credentials is stolen today, it can stay useful for up to 90 days in two of those systems. I'm proposing one identity platform that owns sessions, keys and recovery policy, while product teams keep their signup and onboarding experience. It's about eight engineers for a year, retires two legacy login systems, cuts the maximum lifetime of a stolen credential from 90 days to 10 minutes, and makes our next identity outage a degraded login page instead of a company-wide outage. The main risk is migration friction for the acquired product's users; I'd move them on their next login, not with a forced reset."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "We'll build a secure login with short tokens and MFA" — correct for one product; ignores the four other login systems in the company.
• "Security will decide the recovery policy" — names an owner but not the tension with growth and support, nor the audit.
• "We'll buy an IdP" — no exit plan, no ownership of the token contract, no answer for the vendor's breach.

Appendices

Appendix A: Mechanics in Depth#

A.1 Login and Token Issuance#

A.2 Refresh With Rotation and Grace#

def refresh(rt, dpop_proof):
    sid, mac_ok, exp = parse_and_verify_mac(rt)          # self-verifiable for grace mode
    if not mac_ok or exp < now(): return 401
    try:
        s = session_store.get(sid, timeout=50ms)
    except Unavailable:
        return grace_issue(sid, dpop_proof)               # ≤ 3 × 10 min, deny list checked
    if s.revoked_at: return 401("session_revoked")
    if not dpop_matches(s.binding_key_thumbprint, dpop_proof): return 401
    h = sha256(rt)
    if h == s.current_refresh_hash:
        new_rt = mint_rt(sid)
        ok = session_store.cas(sid, expect=h, set_current=sha256(new_rt), set_prev=h,
                               rotated_at=now())
        if not ok: return refresh_retry_once()
        return issue_access(s), new_rt
    if h == s.prev_refresh_hash and now() - s.rotated_at < 60s:
        return issue_access(s), last_issued_rt(sid)       # idempotent retry, same RT2
    revoke_family(s.family_id, reason="reuse_detected")   # theft or very late replay
    publish("session.revoked", s.family_id)
    return 401("reuse_detected")

A.3 Legacy Password Hash Migration#

Rehash on successful login: verify with the legacy algorithm, then store argon2id and mark the record migrated. For dormant accounts, wrap the legacy hash — argon2id(legacy_hash) — so the weak hash is no longer stored, and verify by computing the legacy hash first. After ~12 months, accounts still on wrapped hashes get a reset on next login.

Appendix B: Data Model and Session Lifecycle#

CREATE TABLE credentials (
  credential_id   TEXT PRIMARY KEY,
  account_id      TEXT NOT NULL,
  type            TEXT NOT NULL,             -- password | passkey | totp | phone | email | federated
  secret_ref      BYTEA NOT NULL,            -- argon2id hash or passkey public key
  added_at        TIMESTAMPTZ NOT NULL,
  trusted_after   TIMESTAMPTZ NOT NULL,      -- cooling-off end; limited mode before this
  revoked_at      TIMESTAMPTZ
);

-- Session store (key-value, TTL = absolute_expires_at)
session:{sid} → { account_id, device_id, family_id, current_refresh_hash, prev_refresh_hash,
                  rotated_at, auth_time, amr, binding_key_thumbprint, expires_at,
                  absolute_expires_at, revoked_at }
account_cutoff:{account_id} → revoked_before timestamp   -- for revoke-all

Refresh tokens are stored only as hashes; a session-store dump does not yield usable tokens.

Appendix C: Coordination Mechanisms#

C.1 Revocation Fan-Out#

The session-store write and the event are coupled through an outbox or change-data capture, so a revocation can't be committed without being published. Event-bus mechanics: Apache Kafka.

C.2 Quick Comparison#

Appendix D: API Contract & Client Behavior#

• Clients refresh at ~80% of access-token lifetime with ±10% jitter; on failure, exponential backoff from 1s capped at 60s, and honor Retry-After.
• Refresh tokens are stored in platform secure storage (Keychain, Keystore) or HttpOnly, Secure, SameSite cookies on web — never in localStorage.
• First-party mobile clients generate a non-exportable device key and send DPoP proofs on refresh.
• 401 session_revoked → clear tokens, go to login. 401 reuse_detected → same, plus show a security notice.
• Login and recovery responses never reveal whether an identifier exists (uniform responses and timing).
• Authorization code with PKCE for every OAuth client; no implicit grant, no password grant, per RFC 9700.

Appendix E: Observability#

Core metrics:

• login.attempts_rate, login.success_rate, login.hash_pool_utilization, login.challenge_rate{device}
• refresh.rate, refresh.error_rate, refresh.grace_issued_total, refresh.reuse_detected_total{client_version}
• revocation.propagation_p99, revocation.canary_accept_after_revoke_total (must stay 0), denylist.entries
• recovery.started_rate, recovery.cancelled_by_owner_rate, recovery.success_then_dispute_rate{channel}
• tokens.verified_with_unknown_session_total (forgery signal), jwks.verifiers_missing_active_kid
• otp.sms_sent{country}, ato.confirmed_per_million_logins

Critical alerts:

Debugging the silent failure: a revoked session that still works produces no errors anywhere. The canary is the only reliable detector; second best is tokens.used_after_revocation computed offline from gateway logs joined to revocation events.

Appendix F: Scale Evolution#

What you don't build on day one: your own HSMs, a central relationship-based authorization service, DPoP for every client, regional identity cells, cross-vendor shared signals. Each has a trigger in Section 11.

Appendix G: Service-to-Service Identity, Multi-Tenancy and Cost#

• Workload identity: certificates live ~12 hours and renew at half-life, so a CA outage of up to ~6 hours doesn't break running services; it blocks new workloads. The CA is tier 0 for deploys, not for traffic. User identity rides alongside as a separate signed context, never as a workload certificate. Platform mechanics: Kubernetes; service registration: Service Discovery.
• Tenant isolation: per-tenant rate limits on SCIM and federation metadata fetches; per-tenant session policy; one tenant's misconfigured IdP produces errors only for that tenant.
• Cost attribution: tag SMS sends and step-up challenges with the requesting product; products that pick SMS as their step-up method see the bill.
• Abuse costs are bursty: budget hash-pool headroom and SMS spend for attack weeks, not average weeks; report cost per 1,000 logins monthly.