How AI Voice Agents Work: The Complete Guide

A voice agent is software that can hold a real-time, two-way phone conversation with a human — understanding what they say, reasoning about it, and responding with natural-sounding speech. Unlike traditional IVR systems that funnel callers through rigid menu trees ("Press 1 for billing..."), voice agents handle freeform dialogue. They understand intent, maintain context across turns, and take actions mid-conversation.

The technology became practical in 2023-2024 when three trends converged: LLMs got fast enough for real-time conversation (sub-500ms to first token), streaming speech-to-text hit near-human accuracy, and neural text-to-speech became indistinguishable from human voices at latencies under 200ms. By 2025, platforms like Vapi and Retell made it possible to deploy production voice agents without building the entire stack from scratch.

Today, voice agents handle millions of phone calls daily — booking appointments, qualifying sales leads, providing customer support, conducting surveys, and even making outbound calls. The market is growing because, unlike chatbots, voice agents meet people where they already are: on the phone.

Every voice agent, regardless of the platform or framework, follows the same five-stage pipeline. A phone call comes in, the audio is transcribed to text, an LLM processes the text and decides what to say, that response is converted back to speech, and the audio is sent back to the caller. This loop runs continuously for every conversational turn.

The critical insight is that this pipeline runs in a loop for every turn of the conversation. A two-minute phone call might involve 15-20 turns, and each turn must complete the full pipeline in under a second to feel natural. This tight latency constraint shapes every architectural decision in voice AI.

Let's walk through each stage in detail.

The telephony layer is the bridge between the phone network and your software. When someone calls your voice agent's phone number, the telephony provider answers the call, establishes a media stream, and sends raw audio data to your application in real time.

Twilio is the most common choice. Their Media Streams API opens a WebSocket connection and streams audio as mulaw-encoded 8kHz chunks — the standard telephone audio format. You get a bidirectional pipe: audio flows in from the caller, and you send synthesized audio back out. Twilio handles call routing, phone number provisioning, SIP trunking, and PSTN connectivity.

Vonage (formerly Nexmo) is the main alternative. Their WebSocket API works similarly but uses linear PCM encoding by default, which is slightly higher quality. Some teams prefer Vonage for its international number availability and slightly lower per-minute rates in certain regions.

For web-based voice agents (no phone number needed), WebRTC replaces SIP/PSTN entirely. The browser captures microphone audio and sends it over a peer-to-peer or server-relayed connection. WebRTC offers higher audio quality (16kHz+ wideband) compared to telephone audio (8kHz narrowband), which downstream STT models benefit from. Platforms like Vapi support both phone and WebRTC simultaneously.

The STT engine converts raw audio into text that the LLM can process. This is the first transformation in the pipeline, and its speed directly impacts the total response latency. Every millisecond the STT takes is a millisecond the user is waiting in silence.

Streaming vs Batch is the fundamental architectural choice. Batch STT processes an entire audio clip after the user finishes speaking — simpler to implement but adds the full transcription time to latency. Streaming STT processes audio continuously and returns partial transcripts as the user speaks, with a final transcript when they stop. For voice agents, streaming is non-negotiable because it lets downstream components start processing before the user finishes talking.

Leading STT Providers

For a deeper comparison, see our Deepgram vs Whisper comparison. The short version: use Deepgram for latency-critical real-time applications, Whisper for accuracy-critical or multilingual use cases.

The LLM is the "brain" of the voice agent. It receives the transcribed text, the conversation history, a system prompt defining the agent's persona and instructions, and optionally context from external systems (CRM records, knowledge bases). It then generates a text response that will be synthesized into speech.

For voice agents, LLM selection is dominated by one factor: time to first token (TTFT). Unlike chatbots where a 2-second delay is acceptable, voice agents need the first token in under 300ms to avoid awkward silence. This pushes most teams toward the fastest available models, even at the expense of raw reasoning capability.

Popular LLM Choices

Function Calling: How Agents Take Actions

Raw conversation is only half the story. Voice agents become truly useful when they can take actions: book an appointment in your calendar, look up an order in your database, transfer the call to a human agent, or send a follow-up text message. This is done through function calling (sometimes called "tool use").

The LLM is given a list of available functions with descriptions and parameter schemas. When the conversation reaches a point where an action is needed, the LLM outputs a function call instead of a text response. The orchestrator executes the function (an API call, database query, etc.), feeds the result back to the LLM, and the LLM generates a natural language response incorporating that result.

For example, a dental office agent might have functions like check_availability(date, provider), book_appointment(patient_name, date, time, provider), and transfer_to_human(reason). The caller says "I need a cleaning next Tuesday," the LLM calls check_availability, gets back available slots, and responds: "Dr. Smith has openings at 10 AM and 2 PM next Tuesday. Which works better for you?"

Context Management

Voice conversations generate context fast. A 5-minute call might produce 3,000-5,000 tokens of conversation history, plus system prompts, function definitions, and retrieved knowledge base content. Managing this context window is critical for both quality and cost.

Production agents typically use a sliding window approach: keep the full system prompt and recent turns, summarize older turns, and drop function call/response pairs that are no longer relevant. Some implementations use a smaller, faster model to generate turn summaries that compress the context window without losing critical information.

TTS converts the LLM's text response into audio that sounds like a human speaking. This is where the "voice" in voice agent comes from, and voice quality has a massive impact on user perception. A robotic-sounding agent triggers immediate distrust; a natural-sounding one keeps callers engaged.

Modern neural TTS has crossed the uncanny valley. The best models produce speech that is indistinguishable from human recordings in blind tests. The challenge for voice agents is doing this fast enough for real-time conversation. TTS latency — the time from receiving text to producing the first audio chunk — is typically the second-largest contributor to end-to-end latency, after the LLM.

Leading TTS Providers

For a detailed comparison, see our ElevenLabs vs Rime comparison. The decision usually comes down to: do you optimize for voice quality (ElevenLabs) or latency (Rime)?

Streaming TTS: The Key to Low Latency

Just like STT, TTS must be streaming for voice agents. Instead of waiting for the entire response to be synthesized (which could take 1-3 seconds for a long sentence), streaming TTS starts producing audio as soon as the first few words are ready. The orchestrator begins playing audio to the caller while the rest of the response is still being generated.

This interleaving of LLM generation and TTS synthesis is what makes sub-1-second response times possible. The LLM streams tokens, the TTS converts them to audio in chunks, and the telephony layer plays them back — all concurrently. Getting this pipeline right is the core engineering challenge of voice AI.

Orchestration is the glue that connects every other layer. The orchestrator manages the real-time flow of data between telephony, STT, LLM, and TTS — routing audio, managing conversation state, handling interruptions, and recovering from errors. It's the most complex part of the stack and the primary reason voice agent platforms exist.

What the Orchestrator Does

• Turn-taking management. Deciding when the user has finished speaking and when to start/stop playing the agent's response. This involves coordinating STT endpointing signals, VAD output, and TTS playback state.
• Interruption handling (barge-in). When the user starts speaking while the agent is talking, the orchestrator must immediately stop TTS playback, discard any queued audio, capture the user's new input, and route it to the LLM with updated context. Poor interruption handling is the most common complaint about voice agents.
• Audio buffering. Smoothing out the inherently bursty nature of streaming TTS. The orchestrator maintains an audio buffer to prevent gaps (buffer underrun) while avoiding adding unnecessary latency (buffer overrun). This is a tuning exercise — different network conditions and TTS providers require different buffer strategies.
• Function execution. When the LLM outputs a function call, the orchestrator executes it (calling external APIs, querying databases), handles timeouts and errors, and feeds results back to the LLM. During execution, it may play filler audio ("Let me check that for you...") to avoid dead air.
• Error recovery. When an STT provider goes down, a TTS request times out, or the LLM returns an unexpected response, the orchestrator must gracefully handle the failure — retrying, falling back to alternate providers, or smoothly ending the call.

Orchestration Approaches

Latency is what separates a good voice agent from a frustrating one. In human conversation, the average response gap is 200-300ms. People perceive anything over 1 second as an awkward pause, and anything over 2 seconds as the other person not paying attention or being confused. Voice agents need to hit sub-1-second end-to-end latency to feel natural.

The Latency Budget

Every pipeline stage contributes to the total. Here's a realistic budget for an optimized voice agent:

Optimization Techniques

• Speculative LLM execution. Start sending STT interim results to the LLM before the user finishes speaking. If the final transcript matches, you've saved the entire STT finalization time. If it doesn't, discard and retry — a small waste for a large latency gain on most turns.
• Token-level TTS streaming. Send LLM tokens to TTS as they arrive, not sentence by sentence. Modern TTS APIs accept partial text and start synthesizing immediately, producing audio for "Sure, I can" while the LLM is still generating "help you with that."
• Smaller, faster LLMs. GPT-4o mini and Groq-hosted Llama models achieve 2-5x faster TTFT than full GPT-4o. For most voice agent conversations, the reasoning gap is unnoticeable to callers.
• Geographic colocation. Run your orchestrator in the same region as your providers. Deepgram and most LLMs are US-East or US-West. A 40ms cross-country round trip adds up when you're doing it four times per turn (STT, LLM, TTS, telephony).
• Filler responses. For turns that require function execution (database lookups, API calls), play a natural filler ("One moment while I look that up") immediately from a pre-cached audio clip. This buys 2-3 seconds without the user perceiving dead air.

The biggest architectural decision in voice AI: do you use a platform that handles orchestration for you, or do you build the pipeline yourself? Both are valid — the right answer depends on your team, your use case, and how much of the stack you need to control.

The five-stage pipeline (telephony, STT, LLM, TTS, telephony) has been the dominant architecture since 2023, but it's about to change. Several trends are converging that will reshape how voice agents work.

Speech-to-Speech Models

OpenAI's GPT-4o and Google's Gemini can process audio natively — no separate STT or TTS needed. The model takes in raw audio and produces raw audio, collapsing the five-stage pipeline into three stages (telephony, model, telephony). This eliminates the information loss of text serialization and dramatically reduces latency. As these models mature and become more accessible, the STT+LLM+TTS pipeline will increasingly become the "legacy" approach.

Emotion and Tone Awareness

Current voice agents are tone-deaf: they process text transcripts and miss all the emotional information in the caller's voice. An angry customer and a happy one produce the same transcript. Speech-to-speech models can perceive and generate emotion, enabling agents that match the caller's energy, calm frustrated callers with softer tones, and express genuine-sounding empathy.

Multimodal Agents

Voice won't stay voice-only. Agents will simultaneously handle phone calls while sending text messages, emails, calendar invites, and visual confirmations. A support agent that talks through a diagnosis and simultaneously sends a screen-share link, a knowledge base article, and a follow-up email is the natural evolution. The orchestration challenge scales accordingly.

Cost Curve

The all-in cost of a voice agent minute has dropped from $0.50+ in early 2024 to $0.08-0.15 in 2026. Open-source STT (faster-whisper), LLMs (Llama, Mistral), and TTS (Coqui, Piper) are approaching commercial-quality at a fraction of the cost. By 2027, running a voice agent will cost less than $0.03/minute, making it economical for use cases that were previously impractical — like handling every single inbound call for a small business.