Design with API Gateways — Staff-Level Technology Guide

Why This Matters#

An API gateway is not a product you pick. It is a decision to put one piece of shared code in front of every request your company serves — and then a decision about which policies live in that code and who is allowed to change them. Envoy vs Kong vs AWS API Gateway is the easy part of the conversation. The hard part is that the gateway becomes the single largest shared blast radius in your architecture, and its configuration becomes a file that 40 teams want to edit on the same afternoon.

Gateways show up in nearly every Staff loop because every design has a front door. "Put an API gateway in front" is the most common box candidates draw, and the least examined. The L5 candidate draws the box and lists features: auth, rate limiting, routing, logging. The L6 candidate says "Envoy-based edge tier, stateless, terminating TLS and validating JWTs locally against a cached JWKS; coarse rate limits locally per pod with a global limit for paid tiers; no business logic and no response aggregation — that's a BFF owned by the product team; routes are per-team HTTPRoute objects in their own repos, and a bad route can only break that team's paths." The L7 candidate asks what the org's policy is for what may be added to the gateway, because a gateway with no such policy turns into an enterprise service bus within two years.

The L5 → L6 gap is not knowing what xDS stands for. It is knowing that every feature you put in the gateway is a feature every request pays for, and every config change is a change to every team's availability. The design problem — building a gateway for a specific system — is covered in the API Gateway case study. This page is about the technology choices and how to run them.

The L5 → L6 → L7 Contrast#

The 60-Second Pitch#

"I'd run an Envoy-based gateway tier as the single public entry point. It terminates TLS, validates JWTs locally against cached signing keys, applies per-client rate limits, and routes by host and path to service clusters — adding about 1ms at p50. It's stateless, so it scales horizontally behind an L4 load balancer and an anycast or DNS front. It does not contain business logic or aggregate responses; product teams own BFFs for that. Each team owns its routes as declarative config in its own repo, validated in CI and rolled out to a canary slice of the fleet first, so a bad route breaks one team's paths for a minute rather than the whole site. For the external partner API, where we need API keys, usage plans and a developer portal, I'd put a managed API-management product in front of the same backends rather than build those features."

The Three Intents#

🎯 Staff Move: "I'm designing for the ingress intent: a thin, fast, safe front door. API-product features like keys and billing I'll buy for the partner API only, and aggregation lives in a BFF the mobile team owns — I don't want the gateway team on the critical path of every feature launch."

The Staff Positions#

Architecture & Internals#

Only the internals that change design decisions.

Data Plane vs Control Plane#

Every modern gateway splits into a data plane (the proxies that handle requests) and a control plane (the system that computes and distributes configuration). Most gateway outages live on the boundary between them.

The Request Pipeline (Filter Chain)#

A gateway is a sequence of filters. Each one adds latency, may call out to another system, and may fail. Order matters.

Put cheap rejections first. Reject unauthenticated or over-limit requests before doing anything expensive. A request that fails JWT validation should never consume a global rate-limit call or an upstream connection.

Configuration Models by Product#

Concurrency and Connection Handling#

Envoy, NGINX and HAProxy all use event loops, roughly one worker per core, each handling thousands of concurrent connections. Consequences for design:

• A blocking call in a filter blocks every request on that worker. Plugins (Lua, Wasm, JavaScript) must be non-blocking; a synchronous HTTP call in a Lua plugin is a classic self-inflicted outage.
• Upstream connection pools are per worker. 500 proxies × 16 workers × 10 connections = 80,000 connections into a service. Use HTTP/2 to upstreams or cap pool sizes, or the gateway becomes the service's largest connection storm on restart.
• Long-lived client connections (gRPC, WebSockets) pin to one proxy. Draining for deploys takes as long as the longest connection allows; set a max connection age (e.g., 10–30 min) to keep fleets rebalanced.
• Request buffering costs memory. Buffering large bodies (uploads) at the gateway turns a 2GB upload into 2GB of proxy memory. Stream bodies or send uploads directly to object storage with pre-signed URLs.

Where State Lives#

Gateways should be stateless, but several features need state:

Core Usage — "The Entire Game": Deciding What Goes in the Gateway#

The quality of a gateway design is determined almost entirely by what you refuse to put in it. Every concern has a natural home; the gateway is the right home only for concerns that are uniform across services and need no domain data.

The Placement Test#

🎯 Staff Move: "The gateway gets anything I could configure identically for every service — TLS, token validation, client rate limits, routing, telemetry. If a policy needs to know what an order or a document is, it lives in the service."

Route Ownership as Config#

The Kubernetes Gateway API makes ownership explicit: infrastructure owns the GatewayClass, the platform team owns the Gateway (listeners, certificates, which namespaces may attach), and each application team owns HTTPRoute objects in its own namespace.

# Owned by the platform team
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: public-edge
  namespace: edge-system
spec:
  gatewayClassName: envoy-edge
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      hostname: "*.example.com"
      tls:
        certificateRefs: [{ name: wildcard-example-com }]
      allowedRoutes:
        namespaces:
          from: Selector
          selector: { matchLabels: { edge-access: "public" } }
---
# Owned by the orders team, in its own namespace and repo
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: orders-api
  namespace: orders
spec:
  parentRefs: [{ name: public-edge, namespace: edge-system }]
  hostnames: ["api.example.com"]
  rules:
    - matches: [{ path: { type: PathPrefix, value: /v1/orders } }]
      backendRefs:
        - { name: orders-v41, port: 8080, weight: 95 }
        - { name: orders-v42, port: 8080, weight: 5 }   # canary
      timeouts: { request: 3s }

The design properties this buys: the orders team can canary its own release without a ticket; it cannot claim /v1/payments (CI rejects overlapping prefixes across namespaces); and a malformed HTTPRoute is rejected for that route only rather than blocking the whole gateway's config.

Authentication at the Edge#

Validate tokens locally; never make a network call per request for authentication.

# Pseudocode for the edge authn filter
def authenticate(req):
    tok = parse_bearer(req.headers.get("authorization"))
    if tok is None:
        return reject(401)
    key = jwks_cache.get(tok.header.kid)          # refreshed every 10 min in background
    if key is None:
        jwks_cache.refresh_async()                # rotated key? refresh once, rate-limited
        return reject(401)
    if not verify_sig(tok, key) or tok.exp < now() or tok.aud != THIS_API:
        return reject(401)
    req.headers["x-auth-subject"] = tok.sub       # strip any client-supplied copies first
    req.headers["x-auth-scopes"]  = ",".join(tok.scopes)
    return next_filter(req)

Two details that interviewers probe:

• Strip inbound identity headers. If services trust x-auth-subject, the gateway must delete any client-supplied version before setting it — or any client can impersonate any user.
• Revocation. Locally validated JWTs can't be revoked before expiry. Keep access-token lifetimes short (5–15 min) and, if instant revocation matters, check a small denylist (bloom filter or in-memory set pushed from the control plane) rather than calling the identity service per request.

When you need external decisions (Envoy ext_authz, Kong plugins, AWS Lambda authorizers), cache the decision by (token, route) for 30–300s and set a deadline of ~20–50ms with an explicit failure mode.

Extensibility — Code in Every Request's Path#

Rule of thumb: any extension that runs on every request needs the same review, testing, canarying, and on-call ownership as the gateway itself. A plugin written by a product team for its own convenience is a liability on everyone's hot path.

The Tunable Tradeoff — Centralization vs. Blast Radius#

Every policy you move into the gateway buys consistency and costs coupling and shared risk. Two quantities make this concrete.

Added latency is additive across filters:

gateway_overhead ≈ TLS (amortized) + route match + Σ local filters + Σ remote calls
typical thin Envoy edge:   0.5–1.5ms p50, 2–5ms p99
with global rate-limit RPC (+1–2ms) and ext_authz (+2–10ms): 4–15ms p99

Availability is multiplicative across synchronous dependencies:

A(path) = A(gateway) × A(authz) × A(rate_limit_store) × A(service)
          0.9999      × 0.9995   × 0.999                × 0.9995  ≈ 0.9979
-> ~18 hours/year of failed requests caused by the front door's dependencies
fail-open on rate limit + cached authz:  ≈ 0.9989-0.9994

🎯 Staff Move: "Authentication is fail-static — we keep the last-known signing keys if the identity provider is down, so revocation can lag by up to 15 minutes, which security has agreed to. Abuse rate limits fail-open with a local per-pod limit as a backstop. Only the partner billing quota fails closed, because over-serving it is a contractual problem."

Anti-Patterns — What Kills Gateway Deployments#

1. The Gateway as an Enterprise Service Bus#

It starts with one header transform, then a response field rename for an old client, then aggregation for the home screen, then a pricing rule "just for this campaign." Two years later the gateway has thousands of lines of per-route logic, one team understands it, and every product launch has a gateway ticket on its critical path. Prevention: a written placement policy and a review for any per-route code.

2. One Global Config, Pushed All at Once#

A single YAML file for 2,000 routes, edited by 40 teams, pushed to every proxy simultaneously. One typo — or one regex that matches everything — and the whole edge is down in seconds. Prevention: per-team route objects, CI conflict detection, staged rollout to a canary slice, automated rollback on 5xx rate.

3. Synchronous Calls Without Deadlines#

An authz or API-key lookup per request with the HTTP client's default 30-second timeout. When that dependency slows, every gateway worker fills with waiting requests and the gateway stops accepting new connections. Prevention: 20–50ms deadlines, cached decisions, declared failure mode per dependency.

4. Retry Amplification#

Client SDK retries 3×, gateway retries 3×, service retries its database 3× — up to 27× load during the exact incident where the backend is struggling. Prevention: gateway retries only idempotent methods, once, within a retry budget (~10% of requests); see Circuit Breakers.

5. Regex-Heavy Routing at Scale#

Hundreds of ordered regex routes evaluated linearly per request: CPU grows with route count, and one catastrophic-backtracking pattern can pin a core. Prevention: prefix and exact matches by default, host-based partitioning, regex only with an engine that guarantees linear time (Envoy uses RE2).

6. A Gateway Per Team#

Every team runs its own Kong or NGINX "for autonomy." Now there are 25 front doors with 25 TLS configurations, 25 ways of validating tokens, and 25 places a CVE patch must land. Prevention: one data plane, shared control plane, per-team route ownership — autonomy in config, not in infrastructure.

7. Buffering Large Bodies#

Uploads and large downloads proxied and buffered through the gateway. Memory spikes, slow clients hold workers, and the gateway's capacity plan is now driven by file sizes. Prevention: stream, set body-size limits (e.g., 10MB default), and route large uploads directly to object storage via pre-signed URLs — see Handling Large Blobs.

8. Treating the Managed Gateway's Limits as Someone Else's Problem#

Building on AWS API Gateway and discovering in production the regional account throttle (10,000 rps steady with 5,000 burst by default), the 29-second default integration timeout, or the 10MB payload cap. Prevention: list the provider's quotas in the design doc and request increases before launch.

The Technology Landscape — Head-to-Head Comparison#

The market sorts into three families. Mixing them up is the most common product-selection mistake.

Product Comparison#

How to choose in one breath: your own edge and Kubernetes ingress → an Envoy-based implementation of the Gateway API (or NGINX/HAProxy if your team already runs them well). Public or partner API that needs keys, quotas and a portal → an API-management product in front of that ingress. Small serverless backend on one cloud → the cloud's managed gateway, until request volume makes per-request pricing hurt.

🎯 Staff Insight: The Kubernetes Ingress API is frozen; the Gateway API (GA at v1.0 in October 2023) is its successor and is implemented by Envoy Gateway, Istio, Kong, NGINX, HAProxy, Traefik and the major clouds. The community ingress-nginx controller announced its retirement in late 2025. Choosing a Gateway API–conformant implementation makes the vendor a two-way door.

Patterns#

Pattern 1: Two-Tier Front Door (Edge + Internal Ingress)#

The edge tier is global, rarely changes, and owned by the platform team. Cluster ingresses are per region or cell and carry team-owned routes. A bad route push affects one cell's ingress, not the global edge. API management sits in front of the edge only for partner traffic.

Pattern 2: Federated Gateways With a Shared Control Plane#

Large orgs run one gateway fleet per domain (payments, commerce, media) using the same data plane and control-plane tooling. Each domain's fleet has its own capacity, change calendar and on-call; the platform team owns the build, the base policies, and upgrades. This is the answer to "one gateway is a bottleneck" that doesn't become "25 different gateways."

Pattern 3: Backend-for-Frontend Behind the Gateway#

Aggregation, client-specific shapes and partial-failure rules live in a BFF that the client team deploys on its own schedule. The gateway stays generic. GraphQL federation is a variant of this pattern — the router is a BFF, not a gateway concern.

Pattern 4: Edge Gateway + Service Mesh#

The gateway handles north-south traffic (client → platform); a mesh (Istio, Linkerd, or an Envoy-based sidecarless mode) handles east-west (service → service) with mTLS, retries and telemetry. Using the same data plane (Envoy) for both means one set of skills, metrics and failure modes. Don't route internal service-to-service traffic back out through the edge gateway — it adds a hop, a shared bottleneck and confusing blast radius.

Pattern 5: Managed Gateway for Serverless#

AWS API Gateway (HTTP API flavor where features allow) → Lambda. No servers, scale-to-zero, built-in authorizers. Watch: per-request cost above ~1–2B requests/month, cold starts in authorizers, and quota limits. The migration path out is a self-run Envoy/ALB layer pointing at the same functions or containers.

Pattern 6: Progressive Delivery at the Gateway#

Weighted routing (1% → 5% → 25% → 100%), header-based routing for internal testers, and shadow traffic (mirror a copy of requests to a new version, discard responses). Pair weights with automated analysis of error rate and latency per backend version so rollback is automatic. Shadowing writes is dangerous — mirror only idempotent reads, or point mirrors at a sandbox.

Pattern 7: Strangler Migration#

Put the gateway in front of a monolith, then move routes one prefix at a time to new services. The gateway is the switch: /v1/orders/* → new service, everything else → monolith. Rollback is a route change. This is one of the strongest justifications for introducing a gateway in an existing system.

Scaling#

The Numbers#

Sizing Example#

Peak:        120K rps, 70% on keep-alive connections, TLS 1.3, avg 4KB response
Per core:    ~6K rps with authn + local rate limit + access logging
Cores:       120K ÷ 6K = 20 cores busy  -> target 50% utilization = 40 cores
Instances:   10 × 4 vCPU per region, + N+2 for AZ loss and deploys = 12
Handshakes:  30% new connections ≈ 36K handshakes/s spike on reconnect storms
             -> ECDSA certs + resumption, or 36K ÷ ~5K/core = 7+ cores just for TLS
Managed alt: avg ~50K rps × 2.6M s/month ≈ 130B requests/month
             × ~$0.90–1.00 per million ≈ $120–130K/month (HTTP API pricing)
             vs self-run ≈ $3–6K/month compute per region + people

At high volume, per-request pricing is the decisive factor; at low volume, people are. The crossover for many teams lands somewhere around a few hundred million to a couple of billion requests per month.

Scaling Moves in Order#

1. Remove work from the hot path — cache authz decisions, local rate limiting, kill unneeded filters, drop full request-body logging.
2. Keep connections warm — client keep-alive, TLS resumption, HTTP/2 to upstreams.
3. Scale horizontally — stateless proxies behind L4; autoscale on CPU and active connections, not just rps.
4. Partition the fleet — per-region and per-cell gateway fleets so one noisy tenant or bad config has a bounded blast radius.
5. Push work to the edge — WAF, bot filtering, and cacheable responses at the CDN before they reach your gateway.

Failure Modes & Recovery#

1. The Bad Config Push#

• Symptom: 5xx or 404 rate jumps across many routes within seconds of a deploy; sometimes every route.
• Root cause: malformed or over-broad route (PathPrefix: / attached to the wrong backend), conflicting routes, a plugin config error, or a control-plane bug sending a partial snapshot.
• Detection: gateway.responses{code=5xx|404} rate by route, correlated with config.version change; synthetic probes on top-20 routes every 10s.
• Fix: automated rollback to last-known-good config version; data plane keeps serving old config if the new one is rejected.
• Prevention: CI validation (schema, overlap detection, "does this route match the health probes of other teams?"), staged rollout to 1% → 10% → 100% of the fleet with automated analysis, per-team route isolation.

t=0       Team merges route change; control plane pushes to all 400 proxies.
t=+4s     Prefix "/" now matches before "/v1/payments"; payments traffic hits the wrong backend.
t=+20s    404 rate on /v1/payments 0% -> 100%. Checkout fails globally.
t=+90s    Synthetic probe alert fires; on-call correlates with config version 8812.
t=+6min   Manual rollback to 8811. Six minutes of failed checkouts.
Fix:      CI rejects routes that shadow another namespace's prefix; staged rollout
          would have contained it to 1% of traffic for ~30s before auto-rollback.
Owner:    Platform team owns the pipeline; the route's team owns the change.

2. Synchronous Dependency Brownout (Authz or Rate-Limit Store)#

• Symptom: gateway p99 climbs from 5ms to seconds; active connections and pending requests per worker climb; eventually connection refusals.
• Root cause: the external authz service or rate-limit Redis slows down; gateway workers wait on it.
• Detection: ext_authz.latency_p99, ratelimit.call_errors, gateway.downstream_rq_active per worker.
• Fix: enforce 20–50ms deadlines; switch to declared failure mode (fail-open for abuse limits with local backstop, fail-static for authz decisions in cache).
• Prevention: decision caching; circuit breaker on the dependency; game-day the dependency's outage.

3. Retry Storm Through the Gateway#

• Symptom: a backend slowing down sees its request rate rise 3–10× instead of fall; recovery stalls.
• Root cause: retries at client, gateway, and service layers multiply; non-idempotent requests retried.
• Detection: upstream_rq_retry rate vs upstream_rq_total; retry ratio > 10%.
• Fix: cut gateway retries to zero for the affected cluster; enable retry budgets.
• Prevention: retry only idempotent methods, once, with a budget (e.g., ≤ 10% of active requests); outlier detection to eject bad hosts rather than retrying into them.

4. Certificate Expiry#

• Symptom: TLS handshake failures for all clients of a hostname at a precise moment; mobile apps with pinned certs fail hardest.
• Root cause: manual renewal missed, or automated renewal failed silently (DNS challenge broken, secret not reloaded).
• Detection: tls.cert_expiry_days per hostname, alert at 21 and 7 days; synthetic TLS checks from outside.
• Fix: emergency renewal and hot reload (Envoy SDS reloads without restart).
• Prevention: automated issuance (ACME or a cloud certificate manager), expiry alerts owned by the platform team, no manual certs in the edge.

5. Connection Exhaustion and Slow Upstreams#

• Symptom: one slow upstream causes errors for unrelated routes on the same gateway.
• Root cause: shared worker or connection-pool resources; slow responses hold connections; no per-cluster concurrency limits.
• Detection: upstream_cx_active and upstream_rq_pending_overflow per cluster; worker event-loop lag.
• Fix: per-cluster circuit breakers (max connections, max pending, max requests), shorter timeouts on the slow cluster.
• Prevention: per-upstream limits as defaults in the route template; bulkheading by fleet partitioning for critical paths.

Operational Reality Matrix#

When to Use vs. Alternatives#

When NOT to Add an API Gateway#

• One backend, one client. A load balancer with TLS and a WAF is enough; a gateway is an extra hop, an extra on-call, and an extra config pipeline.
• To fix internal service-to-service problems. Hairpinning east-west traffic through the edge gateway creates a bottleneck and blurs blast radius. Use a mesh or client libraries.
• As the place to "quickly" add business logic. If the reason is "the service team is too slow," the gateway will become the slow team.
• When the real need is a CDN. Caching, DDoS absorption, and static content belong at the edge network, ahead of your gateway.

Operational Concerns#

What the On-Call Actually Does#

• Correlates error spikes with config versions and rolls back — this is the most common gateway page.
• Distinguishes "gateway is broken" from "a backend is broken and the gateway is the messenger" using per-cluster upstream metrics. Most 5xx at the gateway are upstream 5xx.
• Watches certificate expiry dashboards and renewal job health.
• Handles abusive clients: identifies by key/IP/ASN, applies targeted limits or blocks at the CDN/WAF.
• Runs data-plane upgrades: canary instances first, then by AZ, with connection draining.

Key Metrics & Alerts#

Config Deployment Pipeline#

Upgrades#

Data-plane upgrades are frequent (security fixes in proxies are common) and should be boring: immutable images, canary instances, AZ-by-AZ rollout, connection draining with a bounded max connection age. Keep the control plane and data plane within the version skew the vendor supports, and never upgrade both in the same change.

Interview Application — Staff-Level Plays#

Which Case Studies Use API Gateways#

Every System Design Question Has a Gateway Moment#

• Any public API: "The gateway terminates TLS, validates the JWT locally and applies per-client limits. Authorization on the actual resource happens in the service."
• Mobile app with many screens: "Aggregation goes in a mobile BFF behind the gateway — the mobile team owns it and ships on their own cadence."
• Migration off a monolith: "The gateway is my strangler switch — I move one path prefix at a time, and rollback is a route change."
• Partner/public developer API: "I'll buy API management for keys, usage plans and the portal, and put it in front of the same ingress our own apps use."

What Interviewers Probe#

Common Interview Mistakes#

L5 vs L6 vs L7 Responses#

The Staff API Gateway Checklist#

1. "The gateway is thin: TLS, authentication, scope checks, client rate limits, routing, telemetry."
2. "Authorization on resources lives in the services; aggregation lives in BFFs owned by client teams."
3. "Every synchronous dependency has a 20–50ms deadline and a declared fail-open, fail-closed or fail-static mode."
4. "Retries happen once, at the gateway, for idempotent requests only, inside a retry budget."
5. "Routes are owned per team, validated in CI, and rolled out to a canary slice with automatic rollback."
6. "The platform team owns the data plane, base policies, certificates and upgrades; the page goes to them for gateway-generated errors and to the service team for upstream errors."

🎯 Staff Insight: Say what the gateway won't do. "No business logic, no response composition, no fine-grained authorization, no internal east-west traffic." The refusals are what convince an interviewer you've run one.

Evaluation Rubric#

Strong Hire Signals

Lean No-Hire Signals

Common False Positives

• Knowing every Envoy filter name ≠ knowing what belongs in the gateway.
• A long feature list for the gateway box ≠ a design — it's usually a sign of the ESB trap.
• "We use a service mesh" ≠ having a north-south strategy.

The Principal Lens#

Why L7 Sees This Problem Differently#

At Staff level, the gateway is a component to configure safely. At Principal level it is the place where the org's policies become enforceable — and therefore the place every team will try to put its problems. Security wants WAF rules, finance wants metering, product wants experiments, legal wants regional routing, and each request is reasonable. The L7 job is not choosing Envoy over Kong; it is deciding what the org centralizes at the front door, what it federates, and how a policy gets in or out of the hot path — and pricing the shared risk each addition creates.

The Org-Level Fault Line#

One central gateway platform vs. domain-owned gateways vs. no shared gateway (every team behind its own load balancer).

The Principal default: federated — one data plane (Envoy-class), one control plane and pipeline, Gateway API as the contract, domain-owned fleets and routes, and a short list of org-mandated filters (authn, rate-limit hooks, telemetry, WAF integration). API management is bought, scoped to external API products only.

Cost Model#

Assumptions: self-run on cloud VMs at ~$0.04/vCPU-hr; ~6K rps per core with authn and local limits; 2 regions with N+2 headroom; managed pricing at ~$1.00 per million (HTTP-API-class) or ~$3.50 (REST-API-class); API-management licenses vary widely and are shown as ranges; loaded engineer ~$250K/yr. Directional only.

At startup scale, the managed gateway is cheapest once people are counted. At growth scale, request fees overtake a small platform team. At enterprise scale, the dominant costs are the platform team, cross-AZ traffic between gateway and services, and the cost of outages — which is why cells pay for themselves.

The 3-Year Evolution Path#

One-Way Doors vs Two-Way Doors#

🧭 Principal Move: "I'll let domains choose their fleet sizes, their canary cadence, and even their gateway implementation as long as it speaks Gateway API. I won't let anyone put business logic in a plugin or change what identity headers mean — those are the doors we can't walk back through."

The Standard I'd Write#

RFC-EDGE-003: What Belongs at the API Gateway

Scope: All north-south HTTP/gRPC traffic entering production.

MUST
  1. Enter through a platform-operated gateway fleet; no service exposes a public
     endpoint directly.
  2. Authenticate at the gateway (local token validation); services MUST NOT trust
     identity headers from any other source; the gateway strips inbound copies.
  3. Declare routes as Gateway API resources in the owning team's repo; CI rejects
     overlapping prefixes across teams.
  4. Roll out config changes to <= 5% of the fleet first with automated rollback.
  5. Every synchronous gateway dependency declares a deadline (<= 50ms) and a
     failure mode (fail-open / fail-closed / fail-static), approved by security.
  6. No business logic, response composition, or resource-level authorization in
     gateway filters or plugins.
SHOULD
  7. Retry only idempotent methods, once, within a 10% retry budget.
  8. Use BFFs owned by client teams for aggregation.
  9. Keep partner/public API products behind an API-management layer with a
     published deprecation policy (>= 12 months).

Exceptions: Requested via the edge platform team; approved by the platform lead
and the requesting director; reviewed every two quarters; custom filters require
an owning team on the gateway on-call rotation.

Success metrics: zero Sev1s caused by config pushes; gateway-added p99 < 10ms;
custom filter count flat or decreasing; route change lead time < 1 hour.

What I'd Tell the VP#

"Every customer request passes through our API gateway, so it is both our best place to enforce security and our biggest single point of failure. Two of our last three major outages were gateway configuration changes, not broken servers. I'm proposing three things: each team manages its own routes with automatic safety checks and gradual rollout, the gateway is split into independent sections so one mistake can't take everything down, and we publish a clear rule that product logic stays out of the gateway. That's about two engineers for two quarters. It reduces our most common cause of site-wide outages and stops the gateway team from becoming a bottleneck for every launch."

Principal Interview Signals#

Staff answers that L7 interviewers find insufficient:

• "Use canary rollouts for config" — without cells, so a successful bad push still reaches everyone eventually.
• "Keep the gateway thin" — as a preference, not as a published policy with an exception process.
• "Managed is simpler" — without pricing per-request fees at the org's actual volume.

In the Wild#

Lyft — Envoy#

Lyft built Envoy to give a polyglot microservice fleet consistent networking — retries, timeouts, circuit breaking, outlier detection, and observability — without re-implementing them in every language, and open-sourced it in 2016. It became a CNCF graduated project and the data plane under many gateways and meshes, including Istio, Envoy Gateway, and several commercial products. Its xDS APIs became a de facto standard for dynamic proxy configuration.

Staff insight: Envoy's justification was organizational — inconsistency across teams — not raw speed. Make the same argument for a gateway: it pays for itself by making policy uniform, so keep in it only what should be uniform.

Netflix — Zuul#

Netflix built Zuul as the front door for its streaming API traffic, using filters for routing, authentication, and resiliency, and later rewrote it as Zuul 2 on an asynchronous, non-blocking Netty core to handle very large numbers of persistent connections more efficiently. Netflix has written about using the gateway for regional traffic steering and for testing via traffic shaping and fault injection.

Staff insight: The move from blocking to non-blocking was driven by connection counts and resilience, not latency. Gateways fail by exhausting concurrency when something behind them slows — design for that first.

The Kubernetes Gateway API — Ownership as a Spec#

The Kubernetes SIG-Network community designed the Gateway API (v1.0 GA in October 2023) to replace the limited Ingress resource. Its central idea is role separation: infrastructure providers own GatewayClass, cluster operators own Gateway (listeners, certificates, which namespaces may attach routes), and application developers own HTTPRoute and similar resources. Many vendors implement it, which makes the data-plane choice largely portable.

Staff insight: The spec encodes the ownership model Staff candidates should describe anyway — platform owns the door, teams own their routes. Citing it shows you think about who can change what, not just what the box does.

Practice Drill#

Prompt: "A 300-engineer company has 60 services behind a single NGINX config file maintained by one infra team. Changes take three days, two outages last quarter were config mistakes, and the business is about to launch a public partner API with paid tiers. What do you do over the next two quarters?"

Quick Reference Card#

Job:              thin front door — TLS, authn, scope check, client limits, routing, telemetry
Not its job:      business logic, aggregation (BFF), resource authz, east-west traffic (mesh)
Families:         proxies (Envoy, NGINX, HAProxy, Traefik) | API mgmt (Kong, Apigee, Azure APIM)
                  | cloud-managed (AWS API Gateway, Google API Gateway)
Standard:         K8s Gateway API v1 (GA Oct 2023): GatewayClass / Gateway / HTTPRoute
Added latency:    self-run thin ~0.5–1.5ms p50, 2–5ms p99; managed ~10–30ms
Throughput:       ~5–20K rps per core (lighter filters = higher)
TLS:              ~1–2K full RSA handshakes/s/core; prefer ECDSA + resumption + keep-alive
AuthN:            local JWT verify (~50–200µs), JWKS refresh ~10 min, fail-static
Dependencies:     deadline 20–50ms; declare fail-open / fail-closed / fail-static
Retries:          idempotent only, once, budget <= 10%
Config:           per-team routes, CI overlap checks, 5% canary, auto-rollback
AWS API GW:       10K rps / 5K burst per account-region default; 29s integration timeout;
                  10MB payload; REST ~$3.50/M, HTTP ~$1.00/M
Long connections: max connection age 10–30 min for gRPC/WebSockets

RED FLAGS
  - Aggregation or business rules in the gateway
  - Per-request call to the auth service with no cache or deadline
  - One global config file pushed to the whole fleet at once
  - Retries at client + gateway + service
  - Large uploads buffered through the gateway
  - A different gateway product per team