Rate Limiting by Tokens

Why requests are the wrong unit

One request might be a 50-token classification and another a 150K-token document analysis, a 3,000× difference in cost and provider capacity. A limit of "60 requests per minute per user" lets a single user send 60 huge requests and use up your entire provider quota.

Providers limit you on requests per minute (RPM) and tokens per minute (TPM), per model, for your whole organisation. Your internal limits must mirror that.

Algorithms

Token bucket. Each tenant has a bucket with capacity C tokens, refilled at R tokens per second. A request is allowed if the bucket holds enough for it, and then the amount is subtracted. It allows short bursts up to C while enforcing an average rate R.

Sliding window counter. Sum the tokens used in the last 60 seconds and allow the request if sum + estimate ≤ limit. It is closer to how providers count, and easy to implement in Redis with sorted sets or bucketed counters.

Either works. What matters is what you count and where you store it: a shared, fast store such as Redis, updated atomically with Lua scripts or atomic increments, because gateway nodes are stateless.

Estimating before the call

You don't know the true token count until the response finishes. Handle it in two phases:

1. Reserve before the call: count input tokens (tokenizer, or chars ÷ 4 plus a margin) plus max_tokens for the output.
2. Reconcile after the call: refund the difference between the reservation and the actual usage reported by the provider.

Setting max_tokens sensibly matters twice: it bounds cost, and it keeps reservations from being wildly pessimistic.

Fair sharing across tenants

With a fixed provider quota shared by many tenants or teams:

• Per-tenant TPM budgets, summing to at most the provider limit (or over-committed carefully, with a global check as well).
• Priority tiers: interactive traffic gets guaranteed capacity, and batch or background jobs use what's left. A background job should never cause 429s for a user-facing chat.
• Queue instead of reject for batch work. Hold requests until budget frees up, rather than failing them.
• Weighted fair queuing when demand exceeds supply, so each tenant gets its share and no single tenant starves the others.

Long-window budgets

Short windows protect provider limits. Long windows protect your wallet:

• Daily or monthly spend caps per tenant, team or API key, in dollars (tokens × price, with cached tokens priced accordingly).
• Alerts at 50%, 80% and 100% of budget, then either a hard stop or a downgrade to a cheaper model.
• Per-user caps on consumer products, to prevent abuse such as scripts farming your free tier.

Telling clients what happened

• Return 429 with Retry-After, and headers showing remaining tokens and requests (as providers do).
• Distinguish "you exceeded your quota" from "we are over capacity", because clients should react differently.
• Give internal SDKs automatic backoff that honours these headers.