Designing a ChatGPT-Style Chat Service

A consumer chat product looks simple: a text box, a model and a stream of tokens. At ten million daily users it is one of the most expensive systems you can build. The design is dominated by one resource, GPU time, and one user-facing number, time to first token. Almost every decision below either cuts GPU work or protects latency when GPUs run short.

1. Clarify the task and the quality bar

Questions worth asking before drawing anything:

• Who are the users, and how many? Assume a global consumer product with 10M daily active users (DAU) and about 15 messages per user per day.
• Which models? Assume two tiers we serve ourselves: a fast mid-size model for most traffic, and a larger 70B-class model for paid users and hard requests. (If we used a hosted API instead, the capacity math turns into a cost and rate-limit problem, covered in API vs self-hosted.)
• Which features are in scope? Multi-turn chat, history, file and image uploads, web search, and "memory" of user preferences.
• What does good mean? Helpful, safe answers that stream quickly. Quality is judged by user feedback and offline eval suites, not by one metric.

Functional requirements

• Multi-turn chat with streamed responses; stop, regenerate and edit a previous message (which branches the conversation)
• Conversation history: list, search, resume, rename, share and delete
• Uploads: documents and images, used as context for the conversation
• Tools: web search and a code interpreter, invoked by the model
• Saved memory: user-approved facts ("I'm vegetarian", "I write in British English")
• Plans: a free tier with limits, and paid tiers with higher limits and better models

Non-functional requirements

• 10M DAU, 150M messages a day, with a 3× daily peak
• Time to first token (TTFT) under 1 s at p95 for plain messages; at least 30 tokens/s while streaming
• 99.9% availability for chat. Degrade under load instead of failing
• Safety moderation on input and output; abuse and fraud resistance
• Privacy: deletions honoured everywhere; regional data residency for some markets

Out of scope: training the models themselves, the billing system, and the mobile apps' offline mode.

2. Choose the operating point

Consumer chat is latency-sensitive and cost-dominated. Quality matters, but most messages are everyday tasks (rewrite this, explain that, plan a trip) that a mid-size model answers well. So:

• Stream everything. Perceived latency is TTFT, not total generation time.
• Default to the efficient model, and route to the large model only for paid users or requests that need it.
• Keep context tight. Every extra input token costs prefill compute on every turn.
• Spend quality budget where users notice it: hard reasoning, coding and long documents.

3. Estimates

GPU capacity. Suppose everything ran on the 70B-class model at ≈ 1,500 usable output tokens/s per 2-GPU replica: 2.6M ÷ 1,500 ≈ 1,700 replicas, or ≈ 3,500 GPUs at peak. Now apply model tiering: if 80% of traffic goes to a mid-size model that produces ≈ 5,000 tokens/s per GPU, that share needs ≈ 2.1M ÷ 5,000 ≈ 420 GPUs, and the remaining 20% on the large model needs ≈ 700. Tiering takes the peak from ≈ 3,500 GPUs to ≈ 1,100.

Cost. Average demand is about a third of peak (≈ 370 GPUs), but scaling lag, per-pool minimums and warm spares keep roughly 900 provisioned on average. At an assumed $2.50 per GPU-hour: 900 × 24 × $2.50 ≈ $54K a day, ≈ $1.6M a month, or roughly $0.0004 per message. Free users sending 15 messages a day cost about half a cent each, which is why free-tier limits and model choice are product decisions as much as engineering ones.

4. Architecture

Client

web / mobile, SSE

Edge + API gateway

auth, plan limits, WAF

Chat service

conversation state

Context builder

history, memory, files, tools

Model router

tier, replica affinity

Inference pools

vLLM / SGLang per model

Components

The lifecycle of one turn

1. The client sends the message over HTTPS and opens a server-sent events (SSE) stream. The gateway authenticates and checks the plan's token budget.
2. The chat service loads the conversation's recent turns and summary from cache (falling back to the store) and persists the user's message.
3. The context builder assembles the prompt within a token budget, and moderation checks the input in parallel.
4. The router picks the model and a replica, hashing on conversation ID so the previous turn's KV cache is likely still warm.
5. The model streams tokens. If it emits a tool call (for example web_search), the tool runtime executes it, the result is appended to the context, and generation resumes.
6. Tokens flow back through the chat service to the client. Output moderation samples the stream and can cut it off.
7. When the stream completes, the chat service writes the assistant message, records token usage, and enqueues async work: a title for new conversations, a summary refresh every N turns, and memory extraction.

Data model. Conversations are trees, not lists, because editing a message or regenerating an answer creates a branch:

The client renders the path from the root to the currently selected leaf. Storing parent_id makes branches cheap and keeps regenerate idempotent.

Streaming API. POST /conversations/{id}/messages returns an SSE stream of delta, tool_call, tool_result and a final done event with usage. Each event carries a sequence number, so a client that reconnects after a network blip can resume from the last event it received instead of regenerating. See Streaming responses with SSE.

5. Deep dives

Deep dive A: inference efficiency

This is where the GPU bill is won or lost.

• Prefix caching with affinity. Turn n's prompt is turn n−1's prompt plus the last answer and the new message. Route a conversation to the same replica with a consistent hash on conversation ID, so its KV prefix is warm. Cache hit rates of 70–90% on input tokens are realistic, which cuts prefill compute and TTFT dramatically. The system prompt is shared by every conversation and is always cached.
• KV cache tiering. A user who pauses for five minutes has a KV prefix sitting in GPU memory, doing nothing. Engines can offload idle prefixes to CPU memory or local SSD and reload them in milliseconds, instead of recomputing thousands of tokens.
• Continuous batching, with the batch size tuned to the per-user streaming target (≈ 30 tokens/s). Larger batches raise throughput per GPU but slow every stream in the batch. See Batching and throughput.
• Disaggregated prefill and decode. Long prompts (uploaded files) stall decoding for everyone in the batch. Running prefill on separate replicas and shipping the KV cache to decode replicas keeps streaming smooth.
• Quantization and speculative decoding. FP8 weights and KV cache roughly double how many concurrent streams fit per replica. A small draft model proposing tokens that the large model verifies can raise tokens/s per stream by 2× or more on predictable text.
• Separate pools by shape. Long-context and reasoning traffic goes to its own pool, so it doesn't shrink batch sizes for short chats.

KV cache calculator

See how context length and concurrent users eat GPU memory, and how many sequences one GPU can hold.

Model

GPU

GPUs

Weights / KV precision

Context per sequence8K tokensConcurrent sequences16

Weights 15.0 GBKV cache 16.0 GBActivations + runtime 4.65 GBDashed line: usable GPU memory (76.0 GB)

KV cache per token

128 KB

2 × layers × kv_heads × head_dim × bytes

KV cache per sequence

1.00 GB

at 8,192 tokens

Max concurrent sequences

59

at this context length

Memory needed now

35.6 GB

fits

Deep dive B: model routing and tiering

The router decides which model answers, per message:

The classifier itself must be tiny (a few milliseconds) or it eats the latency budget. Track the escalation rate and the quality gap between tiers on your eval set: if the mid-size model wins 95% of everyday prompts, routing is doing its job. More in Model routing and fallbacks.

Deep dive C: context, memory and files

Conversations can run to hundreds of turns, and the model's context window is not a storage system. The context builder fills a fixed budget in priority order:

1. System prompt and safety instructions (always, and cached).
2. Saved memory: a few hundred tokens of user-approved facts.
3. Recent turns verbatim, up to a budget (for example 6K tokens).
4. A rolling summary of older turns, regenerated asynchronously every N turns so it never blocks a response.
5. Retrieved chunks from uploaded files, instead of whole documents.
6. Tool results, truncated or summarised when large.

This keeps average input near 3K tokens instead of growing without bound. It trades some recall of early details for bounded cost and latency, and the summary is where that trade is tuned.

Memory is a small extraction pipeline: after a conversation, a model proposes durable facts, and the product asks the user to confirm them or shows them in settings. Storing facts without user control creates privacy and "creepiness" problems. Deleting a memory or a conversation must remove its derived data too.

Uploads go through extraction (text, tables, OCR for scans), chunking and embedding into a per-conversation vector index. Small files can be included whole; large ones use retrieval, like a miniature version of the enterprise document Q&A design.

Deep dive D: tools and agentic turns

Web search and the code interpreter turn one request into a loop: the model calls a tool, reads the result and continues.

• Web search: the model writes a query; a search API returns results; a fetcher retrieves and cleans the top pages; the answer cites its sources. Treat every fetched page as untrusted input, because it may contain prompt injection.
• Code interpreter: runs in an isolated sandbox (a microVM or gVisor container) with no network by default, CPU and memory limits, and a time limit. Files it produces go to object storage. See Sandboxing and permissions.
• Budgets: cap tool calls per turn (for example 5) and total tokens per turn, so a confused model can't loop.
• UX: stream intermediate states ("Searching the web…") so the user sees progress while TTFT for the final answer grows.

Deep dive E: overload and fairness

A viral moment or a new model launch can triple traffic in minutes, and GPUs cannot be provisioned in minutes.

• Per-plan token limits in a sliding window, enforced at the gateway before any GPU is touched. See Rate limiting by tokens.
• Admission control: when queue time would push TTFT past the SLO, shed or queue free-tier requests first, and route them to a smaller model ("We're seeing high demand, using a faster model").
• Priority lanes: paid and interactive traffic goes ahead of background work such as titles, summaries and memory extraction, which can wait minutes.
• Autoscaling on queue depth and KV-cache usage, not CPU. Keep warm spare capacity, because loading 140 GB of weights onto a new node takes minutes.
• Burst capacity: a secondary region, or a hosted API as overflow for the mid-size tier, behind the same router.

6. Evaluate, safeguard, operate

Offline evaluation. Every model, prompt or routing change runs an eval suite before release: helpfulness and instruction following (scored by an LLM judge calibrated against human ratings), coding and maths benchmarks, safety refusals and over-refusals, and regression sets built from past incidents.

Online evaluation. Thumbs up and down, regenerate rate, edit rate, conversation length, return rate, and A/B tests for model or prompt changes. Roll new models out as a canary (1%, then 10%) with automatic rollback on error or quality regressions.

Safety.

• Input and output moderation classifiers, with categories and thresholds per market.
• Output moderation runs on the stream in chunks, so it can stop a harmful answer mid-way.
• Abuse detection for scripted accounts, jailbreak farms and resale of free-tier access.
• Prompt-injection defences for tools: fetched content is marked as untrusted data, and tools with side effects require confirmation.

Privacy. Deleting a conversation removes it from the store, caches, search index, memory and derived summaries, and it is excluded from training datasets according to the user's settings. EU conversations can be pinned to EU storage and EU inference. Logs store message IDs and token counts, not content, by default. More in PII, compliance and data residency.

Observability. Trace each turn across the gateway, context builder, router, inference and tools. Dashboards per model pool: TTFT and tokens/s percentiles, queue depth, KV-cache usage, prefix-cache hit rate, batch size, error and moderation rates. See Tracing and observability.

Failure modes

7. How the design evolves

• Launch: one model on a hosted API, a single region, a simple sliding-window context, no memory. Measure usage before buying GPUs.
• Growth: self-host the high-volume tier, add prefix caching and affinity, model routing, and rolling summaries.
• Scale: multiple regions, disaggregated prefill and decode, KV tiering to CPU and SSD, speculative decoding, and a dedicated long-context pool.

8. Bottlenecks and follow-ups