Designing an AI Coding Assistant

A coding assistant is three different systems wearing one plugin. Inline completion must answer in about a quarter of a second, a thousand times a day per developer. Chat must understand a repository it can't fit in context. Agent mode edits files and runs commands, for minutes at a time, with real consequences. Each has its own operating point, its own models and its own failure modes, and a good design keeps them separate.

1. Clarify the task and the quality bar

Questions to ask:

• How many developers, and which editors? Assume 1M active developers across VS Code, JetBrains and a CLI.
• Which modes? Inline completions, chat, and an agent that makes multi-file changes and runs tests.
• Where does code live? Mostly on the developer's machine; enterprise customers also connect hosted repositories for indexing.
• Privacy expectations? Enterprises require no retention and no training on their code, region pinning, and some want a self-hosted option.
• Business model? A subscription of roughly $20–40 per developer per month. Serving cost must stay well below it.
• What is "good"? For completions: suggestions developers accept and keep. For agents: tasks completed with passing tests and a mergeable diff.

Functional requirements

• Inline, multi-line completions using fill-in-the-middle (FIM)
• Chat about the codebase: explain, refactor, write tests, find where something is done
• Agent mode: plan, edit multiple files, run builds and tests, iterate, and present a diff for review
• Repository indexing for large codebases, and organisation-level settings

Non-functional requirements

• Inline suggestions within about 300 ms after the debounce, at p90
• Chat TTFT under 2 s; agent tasks in minutes
• No retention of code by default; enterprise controls (exclusions, regions, audit)
• Serving cost per developer comfortably under the subscription price

2. Choose the operating point

Making this split explicit early is the most important move in the interview. It is why the inline path never waits on a frontier model, and why agents never share a queue with completions.

3. Estimates

Inline completions

This is the opposite of chat: prefill dominates. Consecutive requests from one developer share almost all of their prefix (same file, same region), so prefix caching with session affinity reduces 2K tokens of prefill to roughly 100 new tokens per request, or ≈ 3.5M prefill tokens/s at peak.

GPU estimate. A 7B code model at low batch sizes (to keep ≈ 200 tokens/s per stream) might sustain ≈ 5K output tokens/s per GPU, so 1M ÷ 5K ≈ 200 GPUs for decode, plus ≈ 100–150 GPUs of prefill for the uncached tokens: ≈ 350 GPUs at peak, perhaps 200 on average across regions. At $2.50 per GPU-hour that is ≈ $12K a day, ≈ $0.36 per developer per month. Without prefix caching, prefill alone would need several times more.

For comparison, sending uncached context to a small API model at ≈ $0.25 per million input tokens would cost 1B × 2K = 2T tokens a day, ≈ $500K a day. That is why completion models are small, self-hosted and heavily cached.

Chat and agents (assume API pricing of $3 / $15 per M tokens, with cached input at 10%)

Chat works out to ≈ $5.40 and agents to ≈ $2.60 per developer per month on average. But agent usage is heavy-tailed: a power user running 30 tasks a day costs more than their subscription. Agent mode therefore needs metering, budgets and possibly usage-based pricing.

4. Architecture

Inline path

IDE plugin

context gather, debounce, cancel

Edge proxy

auth, nearest region

Completion service

prompt assembly, filters

Inference pool

small code model, FIM

Post-filter

secrets, duplicates, syntax

Chat and agent path

IDE chat / agent panel

Gateway

model routing, budgets

Repo retrieval

symbols + graph + embeddings + grep

Agent loop

plan, edit, run, observe

Sandbox

builds, tests

Components

5. Deep dives

Deep dive A: the inline latency budget

Techniques:

• Cancellation. A new keystroke cancels the in-flight request on the server, freeing its batch slot immediately. Without server-side cancellation, a fast typist generates a queue of useless work.
• Streaming the first line, so the plugin can show something before the multi-line suggestion finishes.
• Stop conditions tuned per context: stop at the end of the current block or statement rather than always generating the maximum.
• Speculative decoding with n-gram lookup, because code repeats nearby text (variable names, patterns). It is cheap and effective here.
• Regional pools close to developers, since network time is a quarter of the budget.
• Client-side caching of recent suggestions for backspace-and-retype patterns, and skipping requests where suggestions are rarely accepted (inside comments, mid-identifier).

See Prefill vs decode for why TTFT and tokens/s behave differently here.

Deep dive B: context selection

A 2K-token budget is tiny compared with a repository, so ranking context is where quality comes from. Candidates, roughly in priority order:

1. Code around the cursor: the FIM prefix and suffix, trimmed to the enclosing function or class.
2. Imports and referenced symbols: signatures and docstrings of functions and types used nearby, from the language server or a tree-sitter index.
3. Recently viewed and edited files: snippets from the files the developer just touched, which strongly predict relevance ("neighbouring tabs").
4. Similar code in the repository: embedding or lexical search for chat and larger budgets.

Order the prompt so the stable parts come first (file path, language, imports, symbol signatures) and the cursor region last. That maximises prefix-cache reuse between keystrokes. A small learned ranker (trained on which context led to accepted suggestions) usually beats hand-tuned heuristics once there is telemetry.

Deep dive C: repository indexing for chat and agents

Chat questions like "where do we validate webhook signatures?" need more than the open file.

• Symbol index: definitions, references and call sites from tree-sitter or language servers. Exact and fast, and the right tool for "find usages".
• Code graph: imports, calls and inheritance, so retrieval can expand from a hit to its callers and callees.
• Embeddings of functions and classes (not arbitrary line windows), plus file-level summaries for "which module does X".
• Lexical search (ripgrep-style) for literal strings, error messages and config keys.
• Incremental updates: re-index only changed files on each commit or save. A monorepo of 50M lines is ≈ 2M functions, so embeddings at 1,024 dimensions in int8 are ≈ 2 GB per repo version, cheap to hold per branch head.

Retrieval combines these like hybrid search, then the model reads the top results. Agents often do better with tools (search_symbols, grep, read_file) than with a single up-front retrieval, because they can follow references step by step.

Deep dive D: agent mode

An agent task ("add rate limiting to the upload endpoint and a test for it") is a loop: plan, read, edit, run, observe, repeat. See The agent loop.

• Tools: read_file, search, edit_file, run_command (build, test, lint), and optionally MCP servers for issue trackers or docs. See MCP.
• Edit application. Asking the model to rewrite whole files is slow and error-prone. Use search-and-replace blocks or unified diffs, validated before applying; if an edit doesn't apply cleanly, return the error to the model. Some systems use a small, fast "apply" model to merge a sketch of the change into the file.
• Tests as ground truth. The agent runs the relevant tests after editing and must show them passing, or explain why not. This is the strongest guard against plausible but wrong code.
• Budgets. Cap steps (for example 50), tokens, wall-clock time and cost per task. Context grows with every tool result, so truncate large outputs, summarise old steps, and rely on prompt caching. See Controlling agent cost.
• Checkpoints. Snapshot the workspace before each edit so the user (or the agent) can roll back. Long tasks need durable state to survive a crashed worker.
• Human in the loop. The agent proposes a diff; the developer reviews it. Pushing, deploying or running destructive commands always requires approval.

How agent context and cost grow

Every step resends the whole conversation so far: system prompt, tool definitions, and every previous tool call and result.

Model

Agent steps (LLM calls)20System prompt + tool definitions6K tokensTool result added per step1.5K tokensModel output per step300 tokens

Prompt caching

Earlier context billed at the cached rate

Trim tool results

Keep ≤ 300 tokens per result

Cost per task

$0.22

Input tokens processed

462K

3.9× a non-growing context

Final context size

40.2K tokens

4% of the window

At 10K tasks a day

$67,428/mo

Total input grows roughly with the square of the step count, because step n resends everything from steps 1 to n−1. Caching and trimming tool output are the two biggest levers. Capping steps is the third.

Deep dive E: sandboxing, privacy and security

• Sandbox: each agent task runs in an ephemeral container or microVM (Firecracker, gVisor) with CPU, memory and time limits, a network allow-list (package registries only), and no production credentials. See Sandboxing and permissions.
• Prompt injection through the repo. READMEs, comments, issue text and dependency files are untrusted input. A malicious comment can ask an agent to exfiltrate environment variables. Keep secrets out of the sandbox, restrict network egress, and require approval for commands outside an allow-list.
• Hallucinated dependencies. Models sometimes invent plausible package names, and attackers register them. Check new dependencies against the registry and an allow-list before installing.
• Secrets: don't send files that look like credentials as context, and block suggestions containing keys.
• Licence and duplicate filters for suggestions that match public code verbatim, where customers require it.
• Privacy: no retention of prompts or suggestions by default, only aggregated telemetry; region pinning; repository and path exclusions; and a self-hosted deployment for the strictest customers.

6. Evaluate, safeguard, operate

Online metrics

• Acceptance rate of shown suggestions, and retention: the share of accepted characters still present after 5 and 30 minutes. Acceptance alone rewards suggestions that are accepted then deleted.
• Latency percentiles, cancellation rate, and suggestions shown per hour.
• For agents: task completion rate, tests passing, diffs accepted without edits, cost and steps per task.

Offline evaluation

• Completion: held-out repositories with masked spans, scored on exact match and, better, on unit tests passing for generated function bodies.
• Agents: a benchmark of real tasks (issues with the fix's tests hidden), scored by tests passing, in the style of SWE-bench. Refresh tasks regularly, because public benchmarks leak into training data.
• A/B tests for every model or context change, because small offline gains often don't show up online.

Observability. Per-region pools: TTFT, tokens/s, queue depth, prefix-cache hit rate, cancellation rate. For agents: traces of every step and tool call, which make up the debugging record.

Failure modes

7. How the design evolves

• Launch: inline completions from a hosted small model and chat over open files.
• Growth: self-hosted regional completion pools with prefix caching; repository indexing; enterprise privacy controls.
• Scale: agent mode with sandboxes and budgets, learned context ranking, and models distilled from larger ones on accepted-suggestion data. See Distillation.

8. Bottlenecks and follow-ups