Designing an LLM Gateway

Once a company has more than a handful of AI features, every team rebuilds the same things: provider keys in config files, retry loops, token counting, cost spreadsheets, and a PII filter someone wrote in a hurry. An LLM gateway centralises them. It's the platform-engineering design problem of GenAI, and the right framing is reliability and governance infrastructure, not a clever proxy.

1. Clarify the task and the quality bar

Questions to ask:

• Who are the clients? Assume 5,000 engineers and ≈ 300 internal apps: chat assistants, batch document processing, agents, embeddings pipelines.
• Which models? Several hosted providers plus self-hosted open-weight models on internal GPU clusters.
• What must it enforce? Per-team budgets, data policies (some apps may only use EU endpoints or approved providers), and logging with redaction.
• Build or buy? Open-source and commercial gateways exist (LiteLLM, Envoy- and Kong-based AI gateways, cloud offerings). Assume we build on an existing proxy core but own the policy, quota and routing layers, because they encode company rules. See Build vs buy.
• What is "good"? Apps never notice it, except that they stop being paged for provider outages and finance stops asking who spent what.

Functional requirements

• One OpenAI-compatible API (chat, embeddings, batch, tool calling, streaming) for all models
• Per-app credentials; provider keys never leave the platform
• Quotas per app and team in tokens per minute and dollars per month
• Model aliases, routing, retries and failover
• Logging and tracing with PII redaction; cost attribution and dashboards per team
• Guardrail hooks (PII, injection screening, content policy) configurable per app
• Opt-in caching; capture of sampled traffic for eval datasets

Non-functional requirements

• ≈ 2K QPS at peak today, growing 3× a year
• Under 10 ms of added p99 latency, excluding guardrail model calls
• 99.99% availability, multi-region
• Tens of thousands of concurrent streams

2. Choose the operating point

Reliability and latency overhead are the constraints; governance (cost, data policy, visibility) is the value. The gateway's own compute is cheap compared with the model spend flowing through it, so there is no reason to economise on redundancy.

3. Estimates

A 1% improvement in routing or caching is worth more than the whole gateway's infrastructure budget, which is why cost features pay for the platform.

4. Architecture

Components

Request lifecycle

1. The app calls POST /v1/chat/completions with its credential and model: "chat-default".
2. The node authenticates the app (a platform-issued key or workload identity), loads its policy from the local config cache, and runs pre-request guardrails configured for it.
3. It estimates tokens and reserves quota for the app and team.
4. The router resolves chat-default to a candidate list filtered by the app's data policy, picks the healthiest candidate, and forwards the request with the platform's provider key.
5. Streamed chunks pass straight through to the client. The final usage event reconciles the reservation.
6. A usage event (app, team, model, tokens by type, latency, cost, status) is published asynchronously, and a sample of redacted content is logged.

5. Deep dives

Deep dive A: token quotas across stateless nodes

A central counter hit on every request adds latency and becomes a hotspot. The design:

1. Estimate the request's tokens: the input count plus max_tokens.
2. Reserve atomically in Redis (a Lua script over a sliding-window counter per app and per team).
3. On completion, reconcile with the actual usage from the provider's final usage event, releasing the unused reservation.
4. Local leasing for high-QPS apps: a node leases a chunk of budget (for example 50K tokens) and spends it locally, which cuts Redis round trips at the cost of slight over-admission.
5. Fail open or closed by tier: if Redis is down, critical apps fail open with local limits, and batch apps fail closed.

Monthly dollar budgets are computed from usage events (tokens × price per token type) with alerts at 80% and 100%, and optional hard stops or automatic downgrades to a cheaper alias. See Rate limiting by tokens.

Deep dive B: pooling provider capacity

Provider rate limits (tokens and requests per minute) usually apply to the whole company's account, not per app. Without coordination, one team's batch job consumes the shared limit and every other app gets 429s.

• Capacity model: the gateway knows each provider deployment's limits and tracks usage against them per region.
• Allocation by priority: interactive, customer-facing apps get guaranteed shares; internal tools get the next tier; batch jobs get whatever is left.
• Batch lane: large offline jobs go through a queue that drains at the spare rate, or to the provider's batch API at a discount. See Batch and async LLM jobs.
• Provisioned throughput: for predictable high-volume apps, reserved capacity with a provider is cheaper and more reliable, and the router prefers it before overflowing to on-demand.

Deep dive C: routing and failover

• Aliases (chat-default, extract-cheap, embed-v3) map to ordered candidate lists with weights and constraints (region, data policy, context length, tool support). Apps depend on aliases, so the platform can change the model behind one after running evals.
• Health scoring from live error rates, 429s and TTFT per provider, model and region. Circuit breakers open on sustained failures, with half-open probing.
• Retries only before the first streamed token, with backoff and jitter, respecting Retry-After. After the first token, the client owns recovery.
• Failover order: the same model in another region, then an eval-approved equivalent model, then degraded mode (a clear error the app can handle).
• Data policy constraints hold during failover: an app tagged "EU data only" never fails over to a US endpoint, even if that means failing.
• Self-hosted routing: for internal clusters, route by queue depth and KV-cache usage (and prefix affinity) rather than round-robin, because request costs vary enormously.

See Model routing and fallbacks.

Deep dive D: streaming at scale

• Async, event-driven nodes (Go, Rust or Envoy-based) hold tens of thousands of concurrent streams each, with little memory per connection.
• Pass chunks through immediately; never buffer a whole response. Count tokens from the final usage event, or tokenize incrementally if the provider doesn't send one.
• Client disconnect cancels the upstream request, which stops the spend.
• Idle and total timeouts per alias; load balancers tuned for long-lived connections; graceful draining during deploys so streams finish.

Deep dive E: caching

• Provider prompt caching is the most valuable and least risky: the gateway keeps stable prefixes (system prompts, tool schemas) first and tracks cached-token pricing, so apps see the savings in their cost reports.
• Exact-match response caching for deterministic calls (temperature 0, same model and prompt), per app, with short TTLs. Useful for embeddings and repeated extraction.
• Semantic caching is opt-in per app for low-risk intents only, with tenant- and permission-scoped keys and quality monitoring. Serving the wrong cached answer to a similar-looking question is a correctness bug. See Prompt caching vs semantic caching.

Deep dive F: guardrails, security and data governance

• Policy plugins in the request path, configured per app: PII detection and masking, prompt-injection screening for apps that process untrusted content, content classifiers on outputs. Run in parallel where possible, with latency budgets, and fail according to the app's tier.
• Redaction before logging, and optionally before sending to providers for apps that require it.
• Data policies: which providers each app may use (contractual zero-retention, regions), enforced by the router.
• Access control on logs by team, with retention limits; an audit log of every config change (who changed routing or budgets, and when).
• Credential hygiene: apps get platform keys that can be revoked individually; provider keys are rotated centrally and never exposed.

See PII, compliance and data residency.

Deep dive G: cost attribution and eval data

• Usage events carry app, team, feature tag, model version, and tokens by type: input, cached input, output, reasoning, and batch. Pricing is applied in the warehouse, versioned, so a price change doesn't rewrite history incorrectly.
• Dashboards per team: spend, tokens, latency, errors, cache hit rate, and the top prompts by cost.
• Anomaly alerts: a sudden 10× in an app's spend usually means a loop or a bug, not success.
• Eval data capture: apps can opt in to sampled, redacted request–response logging, which becomes the raw material for golden sets and regression tests. See Offline evals and golden sets.

6. Evaluate, safeguard, operate

Platform SLOs: availability, added latency (p50 and p99 overhead excluding upstream time), time to detect and route around a provider incident, and quota accuracy.

Model changes behind aliases go through an eval process: each alias has a representative test set, and a candidate model must match or beat the current one on quality and cost before weights shift. Roll out by weight (5%, 25%, 100%) with automatic rollback on error rates.

Availability

• Multi-region active-active; nodes are stateless, so any region can serve any app.
• Redis per region; quotas tolerate brief inconsistency between regions.
• Bypass plan: documented emergency direct-provider credentials for tier-0 apps, used only in a gateway-wide incident.
• Canary deploys, config validation and fast rollback. A bad routing config is a company-wide outage, so config changes get the same rigour as code.

Failure modes

7. How the design evolves

• v1: a thin proxy with per-app keys, logging and cost attribution. Just visibility is valuable.
• v2: quotas, aliases, failover and redaction.
• v3: capacity pooling and priority lanes, caching, guardrail plugins, eval-data capture, and routing to self-hosted clusters.

8. Bottlenecks and follow-ups