Designing a Real-Time Voice Agent

Text chat forgives a two-second pause; a phone call doesn't. Silence longer than about a second makes callers say "hello?", repeat themselves, or talk over the agent, and the conversation falls apart. So voice agents are designed backwards from a latency budget of roughly one second, spent across components that each want more. The second hard problem is conversational: knowing when the caller has finished speaking, and stopping instantly when they interrupt.

1. Clarify the task and the quality bar

Questions to ask:

• Inbound, outbound or both? Assume inbound customer-support calls plus web voice sessions. Outbound calling adds consent and regulatory rules.
• Scale? 10K concurrent calls at peak, average call length ≈ 4 minutes.
• What can the agent do? Answer questions (RAG over policies), look up and change orders (tools), and transfer to a human with context.
• Languages? English and Spanish at launch, including callers who switch between them.
• Telephony? Calls arrive over SIP trunks from a carrier or a CPaaS provider (such as Twilio), plus WebRTC for web and app.
• Compliance? Recording disclosure, PCI rules for card payments, retention limits, and disclosure that the caller is talking to an AI where required.
• What is "good"? Calls resolved without transfer, callers who don't feel they're talking to a slow machine, and no mistakes on actions.

Functional requirements

• Natural conversation with interruptions, questions and actions
• Identity verification before account actions
• Transfer to a human with a summary, and callback scheduling when queues are long
• Recordings and transcripts according to policy

Non-functional requirements

• Voice-to-voice latency (the caller stops speaking → the agent starts speaking) under 1 s at p50 and 1.5 s at p95
• 10K concurrent calls at peak; 99.95% availability during business hours
• Card numbers never reach the LLM, transcripts or recordings

2. Choose the operating point

Latency is the product. Beyond about 1.5 s, callers talk over the agent and the experience breaks. So use a fast model, accept slightly lower reasoning quality, and escalate complex cases to humans. Within the latency constraint, correctness of actions comes next, exactly as in the customer-support agent, which shares its tools and guardrails with this design.

3. Estimates

The latency budget

Load

Cost per minute (order-of-magnitude, varies widely by vendor and whether you self-host)

A human agent costs roughly $0.50–1.50 per minute fully loaded, so even at the high end the economics work. The real constraints are latency and quality. At 110M minutes a month, self-hosting ASR and TTS models becomes worth evaluating, because they are the largest line items.

4. Architecture

Components

The lifecycle of one turn

1. The caller speaks. Audio frames (20 ms each) flow from the media server to the ASR, which emits partial transcripts continuously.
2. The turn manager watches silence and the partial transcript. When the semantic end-of-turn model is confident the caller has finished, it finalises the turn.
3. The orchestrator sends the transcript and state to the LLM, which streams its reply.
4. As soon as the first sentence is complete, it goes to TTS, and the first audio chunk plays within ≈ 100–200 ms. Later sentences synthesise while earlier ones play.
5. If the caller starts speaking while the agent talks, the turn manager stops playback, cancels the generation and records exactly what was spoken.

5. Deep dives

Deep dive A: spending the latency budget

• Stream everything: ASR partials, LLM tokens into TTS sentence by sentence, TTS audio in small chunks.
• Speculative start: begin the LLM call on a confident partial transcript before the turn is officially over, and discard it if the caller keeps talking. This hides most of the endpointing delay at the cost of some wasted tokens.
• Short first sentence: prompt the model to lead with a brief acknowledgement ("Sure, let me look at that."), so TTS starts quickly while the substance follows.
• Prefix caching: the system prompt, tool schemas and conversation so far are a stable prefix, so TTFT stays low as the call goes on. See Prefill vs decode.
• Co-location: ASR, LLM and TTS in the same region (ideally the same cluster) as the media servers. Cross-region hops cost 50–150 ms each.
• Latency-optimised serving: a small or mid-size model at moderate batch sizes, since big batches raise TTFT. See Capacity planning for inference.

Deep dive B: turn-taking and barge-in

• Endpointing: voice activity detection alone cuts people off when they pause mid-thought. Combine silence duration with a semantic end-of-turn model, a small classifier on the partial transcript: "I want to change my…" is not finished; "I want to change my address." probably is.
• Adaptive timeouts: allow longer pauses when the caller is reading out a number or thinking ("um, let me find it").
• Barge-in: when the caller speaks while the agent is talking, stop TTS within ≈ 100–200 ms, cancel the in-flight LLM generation, and truncate the history to what was actually played, so the model knows what the caller heard.
• Backchannels: "uh-huh", "okay" and "right" are not interruptions. "Wait, no" is. A small classifier on short utterances decides.
• Echo cancellation, so the agent's own voice coming back through the caller's speakerphone isn't transcribed as the caller speaking.

Deep dive C: cascaded vs speech-to-speech

A pragmatic choice: cascaded for transactional support (tools, compliance, auditability), and speech-to-speech for conversational experiences where naturalness matters most. Keep the orchestrator's interface abstract enough to swap between them per use case.

Deep dive D: prompting and tools for speech

Voice changes what a good answer looks like.

• Spoken style: short sentences, no markdown or lists, one question at a time, numbers and dates written the way they should be spoken. Long answers are offered in parts: "There are three options. Want to hear them?"
• Read-back of critical details: "That's order 4-7-2-9-1, delivering to 12 Oak Street. Is that right?" ASR errors on names, addresses and IDs are the most common cause of wrong actions.
• Pronunciation: a lexicon for product names and brand terms, and SSML hints for the TTS where supported.
• Filler for slow tools: "Let me check that order for you…" while a lookup runs. Silence while a tool works feels like a dropped call.
• Prefetch: look up the caller's account and recent orders by phone number at call start, in parallel with the greeting, so the first account question needs no tool round trip.
• Same guardrails as chat: tools act with the verified caller's identity, and business limits live in code. See the customer-support agent.

Deep dive E: ASR quality, identity and sensitive data

• Domain vocabulary: boost product names, plan names and common order-number formats (contextual biasing). Measure word error rate on your own calls, not a benchmark.
• Accents and noisy lines: evaluate per language and accent; route low-confidence transcripts to a confirmation question rather than an action.
• Code-switching: callers switching between English and Spanish need a multilingual ASR model and an LLM prompt that answers in the caller's current language.
• Identity verification: caller ID is spoofable, so account actions need a second factor (a code sent by SMS, or knowledge-based questions). Don't rely on voice biometrics alone, since voice cloning makes them attackable.
• Card numbers: switch to DTMF capture (the caller types on the keypad) handled by a PCI-compliant payment service, and pause recording and transcription during it. The digits never reach the LLM.

Deep dive F: scaling per concurrent call

• 10K concurrent calls means 10K live ASR streams, up to 10K TTS streams, and 1–2K LLM requests per second at peak.
• ASR and TTS run on GPUs with streaming batching. Capacity is measured as concurrent streams per GPU, found by benchmarking at the target latency. Plan pools per model and language.
• LLM: a latency-optimised pool with prefix caching, sized by requests per second at the TTFT target, not by throughput. Keep headroom, because a slow LLM turns directly into dead air.
• Media servers scale horizontally with sticky sessions: all of a call's audio stays on one server.
• Graceful overload: when capacity runs short, offer a callback or route to a traditional IVR instead of degrading latency for every active call.

6. Evaluate, safeguard, operate

Evaluation

• Latency p50 and p95 per stage and end to end, measured from real audio timestamps (when the caller stopped, when the agent's audio started), not from service logs.
• ASR word error rate on your domain, per language and accent.
• Simulated callers: LLM-driven synthetic callers with goals and personalities, speaking through TTS with added noise, accents and interruptions, run against every release. They test turn-taking and barge-in, which text evals can't.
• Task success (resolved without transfer), transfer rate by reason, average handle time, repeat calls within 7 days, and CSAT from post-call surveys.
• Human QA on a sample of calls, with findings feeding the eval set.

Compliance and safety

• Announce recording and, where required, that the caller is speaking with an AI.
• Pause recording during payment capture; redact PII from transcripts; set retention by jurisdiction.
• Outbound calling has its own consent and calling-hours rules, which the dialler must enforce in code.
• Rate-limit and monitor for abuse (callers probing the agent to extract information about other accounts).

Failure modes

7. How the design evolves

• v1: a voice front end on the existing chat support agent, cascaded pipeline, one language, transfer on anything unusual.
• v2: semantic endpointing, speculative starts, prefetching, simulated-caller evals, and a second language.
• v3: self-hosted ASR and TTS for cost, speech-to-speech for suitable call types, outbound use cases with consent handling.

8. Bottlenecks and follow-ups