Realtime voice agents

Verified 2026-08. Designing a system a person can hold a conversation with. The engineering problem is latency and interruption, not model quality.

Two architectures

Cascaded (STT -> LLM -> TTS). Three components in series. You keep the text transcript, can swap any stage, and can run tools and RAG on the text. You pay serial latency and lose paralinguistics — tone, hesitation, emotion — at the first hop.

Speech to speech. A single model takes audio in and emits audio out with no explicit text stage (the realtime APIs from the major providers). Lower latency, preserves prosody and can respond to how something was said. You give up the intermediate transcript for logging and tool routing, and you are locked to one provider’s model.

The pragmatic 2026 answer is cascaded by default, speech-to-speech where naturalness is the product. Cascaded gives you the transcript that compliance, analytics and evaluation all need, and it lets you use your existing tool-calling and RAG stack. Say that, then name the case that would flip you.

The latency budget

Conversation breaks down above roughly one second of response delay, and users start talking over the agent. Budget backwards from that:

Two design consequences follow:

• Overlap everything. Start the LLM on the partial transcript, start TTS on the first sentence of the LLM stream. A serial pipeline blows the budget on its own.
• Endpointing is where the time goes. A fixed silence threshold is either slow (long threshold) or interrupts people who paused to think (short threshold). This is the single biggest lever on perceived responsiveness.

Turn detection and barge-in

The hard human-factors problem, and the part that separates a demo from a product.

Turn detection decides when the user has finished. Naive VAD plus a silence timer treats a mid-sentence pause as the end of a turn. Production systems use a learned turn-taking model that distinguishes a genuine end of turn from a thinking pause, and both frameworks and platforms now ship one (Pipecat’s smart turn analyser, LiveKit’s turn detector, and equivalent endpointing controls elsewhere).

Barge-in is the user interrupting while the agent is speaking. Handling it means:

• Detect user speech during playback (with echo cancellation, or you detect your own output).
• Stop TTS playback immediately and flush the buffer.
• Cancel the in-flight LLM generation so you are not paying for and queueing a reply nobody wants.
• Truncate the conversation history at what was actually heard, not what was generated. This is the subtle one: if the agent generated three sentences and was cut off after one, the model must believe it said one sentence, or every subsequent turn is grounded in words the user never heard.

Distinguish backchannel from interruption. “Mm-hm”, “right”, “yeah” are acknowledgements, not interruptions. Treating them as barge-in makes the agent stop constantly and feel broken. This classification is exactly what the learned turn models exist to do.

Transport

Echo cancellation is the argument people underestimate. Without it, the agent’s own audio comes back through the microphone and it interrupts itself. WebRTC stacks provide it; a WebSocket audio pipe does not.

For telephony, an SIP gateway bridges to WebRTC. The constraint to remember is narrowband 8 kHz audio, which measurably degrades ASR accuracy compared with the wideband audio a browser sends.

Frameworks

Orchestrating this by hand — VAD, endpointing, streaming ASR, LLM cancellation, streaming TTS, interruption, transport — is a lot of machinery. The established options are Pipecat (vendor-agnostic, open source) and LiveKit Agents (built on their WebRTC infrastructure), plus managed platforms that trade flexibility for a faster start.

Choose a framework unless the pipeline itself is your product. The interesting work is in the conversation design, tools and evaluation, not in re-implementing endpointing.

Tools, RAG and state

A voice agent that only talks is a demo. The product needs it to do things — look up an account, book a slot, escalate.

• Tool latency is user-visible. A three-second database query is a three-second silence. Fill it: acknowledge verbally (“let me check that”), or start the lookup speculatively on the partial transcript.
• Confirm before irreversible actions. ASR errors are silent and confident. Read back the important values — amounts, dates, names — before acting on them.
• Keep the state machine explicit. Free-form LLM conversation plus critical transactions is a bad combination; a defined flow with the model handling the language is the reliable pattern.
• Design the escalation path. Every voice agent needs a route to a human, triggered by repeated failure, explicit request, or detected frustration.

Evaluation

Text-agent evaluation does not cover the parts that make voice fail.

• Conversational metrics: turn-taking gap, barge-in success rate, false-barge-in rate (backchannel treated as interruption), and time to first audio.
• Task metrics: completion rate, escalation rate, and containment.
• Replay a corpus of real calls through the pipeline on every change. Prompt and model changes regress interruption handling in ways nobody predicts.
• Test in bad conditions: packet loss, background noise, accents, telephone audio. A pipeline evaluated only on clean browser audio will not survive a call centre.

See ../13_evaluation/.

Interview angle

• “Cascaded or speech-to-speech?” - cascaded by default, because you keep the transcript for logging, evaluation, compliance and tool routing, and you can swap any stage. Speech-to-speech when naturalness and prosody are the product and you accept the provider lock-in and the loss of an intermediate transcript.
• “What is the latency budget for a voice agent?” - roughly a second end to end before conversation degrades. Endpointing is usually the largest single component, so overlap stages: start the LLM on the partial transcript and TTS on the first sentence.
• “How do you handle a user interrupting the agent?” - stop playback and flush the TTS buffer, cancel the in-flight LLM generation, and truncate history at what was actually heard rather than what was generated. That last step is what stops the conversation drifting from reality.
• “Why WebRTC rather than WebSockets for audio?” - latency, plus echo cancellation, jitter buffering and packet-loss concealment. Without echo cancellation the agent hears its own output and interrupts itself.
• “What breaks a voice agent that a text agent does not have to worry about?” - turn detection treating a thinking pause as the end of a turn, backchannel (“mm-hm”) misread as interruption, and confidently wrong ASR on names and numbers. The mitigations are a learned turn model and reading values back before acting.
• “How do you evaluate one?” - replay real call recordings on every change and measure turn-taking gap, barge-in and false-barge-in rates, and task completion - not just whether the text response was good.