Voice AI Agents Explained: How They Work and Who's Building One

Voice AI agents, explained

A voice AI agent picks up a phone call, a web widget, or a SIP trunk and holds a real conversation instead of routing a caller through a keypad menu. The category has moved fast since 2025: model providers ship dedicated realtime APIs, several venture-funded platforms sell the whole stack as a subscription, and one widely used framework gives it away as open source. Anyone evaluating voice AI agents today is choosing between three different things at once: a model architecture, a hosted platform, and a set of guardrails.

Voice AI agents are AI systems that listen to spoken input, reason about what a caller wants, and reply in synthesized speech close to real time. They differ from IVR menus and text chatbots because they handle open-ended questions, interruptions and follow-up requests with a language model instead of a fixed script or a decision tree.

How voice AI agents work

Most voice AI agents run a three-stage pipeline: speech-to-text (STT), an LLM that decides what to say and which tools to call, and text-to-speech (TTS). LiveKit's own developer guide names Deepgram, AssemblyAI and Cartesia as STT providers capable of sub-300ms transcription, and Cartesia, Rime and Deepgram among the TTS options teams wire into that middle stage. The LLM stage is the same model doing the reasoning in a text agent: GPT, Claude or an open model, deciding intent, holding context and triggering function calls.

A newer path skips the three stages entirely. Speech-to-speech (or "realtime") models take audio in and produce audio out directly. OpenAI's voice agents guide lays out three architectures side by side: GPT-Live for full-duplex conversations that hand reasoning off to a separate backend, the Realtime API's RealtimeAgent and RealtimeSession for speech, reasoning and tool use inside one session, and a chained pipeline for teams that want to inspect or transform the transcript between stages. A minimal chained pipeline using OpenAI's Agents SDK looks like this:

from agents import Agent, function_tool
from agents.voice import AudioInput, SingleAgentVoiceWorkflow, VoicePipeline

@function_tool
def get_weather(city: str) -> str:
    return f"The weather in {city} is sunny."

agent = Agent(
    name="Assistant",
    instructions="You are a helpful voice assistant.",
    model="gpt-6-astra",
    tools=[get_weather],
)

pipeline = VoicePipeline(workflow=SingleAgentVoiceWorkflow(agent))

Latency is the constraint that shapes every design choice here. LiveKit's guide puts the ceiling at roughly one to two seconds of total round trip before callers start talking over the agent, and names WebRTC as the transport built for that budget, ahead of WebSockets (500ms to 1.5s) or plain HTTP streaming (2 to 5 seconds). Turn-taking, meaning detecting when a caller has actually finished speaking rather than just pausing, and handling "barge-in" when a caller interrupts mid-reply, are the two hardest problems on top of that transport choice.

Examples: who is building voice AI agents

OpenAI ships three ways to build one: GPT-Live, the Realtime API, and the chained Agents SDK pipeline described above, documented in its voice agents guide.

ElevenLabs sells ElevenAgents, deployable across phone, web, WhatsApp and email from a single configuration, with over 11,000 voices in more than 70 languages and an Expressive Mode that adapts tone to detected frustration or urgency. It connects to Twilio, Genesys, Telnyx or any SIP-compatible PBX, per its voice agents page.

Retell AI offers two agent-building modes, a node-based Conversation Flow Agent and a Single Prompt Agent, plus a Conductor copilot that builds and tests agents, per its documentation. Its per-minute pricing starts around $0.07 with volume discounts above $3,000 a month, and it holds HIPAA, GDPR and SOC 2 Type I and II certifications.

Telnyx runs its own carrier network under the voice agent, claiming sub-200ms response latency and A-level STIR/SHAKEN caller verification, and prices STT, TTS and orchestration together at $0.05 a minute, per its product page.

xAI's Grok Voice Agent Builder, currently in beta, deploys a no-code agent in under two minutes with built-in connectors to Gmail, Calendar, Outlook, Linear and Notion, plus custom MCP support for internal systems, per x.ai/voice.

LiveKit publishes its agent framework as open source, with 14,300 stars on the livekit/agents GitHub repository and 21,000 on the core livekit/livekit repository at the time of writing, and pairs it with a hosted cloud platform for teams that don't want to run the transport layer themselves, per livekit.com/voice-agents.

Rasa Voice takes the opposite approach from most of this list: fully self-hosted, sovereign deployment where the team picks its own ASR and TTS providers rather than a bundled stack, aimed at regulated enterprises, according to Rasa's own comparison post.

When to use it (and when not)

Voice AI agents earn their cost where call volume is high and the questions are repetitive: order status, appointment booking, tier-one support, lead qualification. GetVoIP's testing notes pricing that ranges from usage-based per-minute rates (Bland.ai at $0.11 to $0.14 a minute, Synthflow near $0.13 a minute on top of a monthly platform fee) to flat enterprise contracts (Cognigy averaging roughly $115,000 a year per Rasa's comparison), so the volume math has to work before the switch pays off.

Skip it, or keep a human in the loop, where a wrong answer is costly and guardrails are still maturing. ElevenLabs itself labels its own Guardrails 2.0 system as Alpha, and even its blocking execution mode, which waits for a safety check before the agent replies, only adds 200 to 500 milliseconds and only catches what the check is configured to catch. Anywhere the conversation could plausibly involve account takeover, medical guidance or financial advice needs a defined escalation path to a person, not just a system prompt.

What's new (as of 21 September 2026)

Retell shipped Retell Workflows on August 24, 2026: native orchestration for everything that happens before a call connects and after it ends, an integration library covering Slack, Google Drive and Salesforce, a tool store, and preset functions. The same changelog entry put GPT-5.6 and GPT Realtime 2.1, both new OpenAI models including a low-latency realtime option, live on the platform.

xAI's Grok Voice Agent Builder remains in beta and publishes its own benchmark, the tau-voice Bench leaderboard, where Grok Voice Think Fast 1.0 scores 67.3% overall against Gemini 3.1 Flash Live at 43.8% and GPT Realtime 1.5 at 35.3%. Telnyx continues to advertise sub-200ms response times and $0.05-a-minute all-in pricing covering STT, TTS and orchestration on one stack. ElevenLabs' Guardrails 2.0 is live in Alpha with three layers: a Focus Guardrail that reinforces the system prompt, a Manipulation Guardrail that can terminate a conversation flagged as a prompt-injection attempt, and Content or Custom guardrails that check each reply before it reaches the caller.

Key takeaways

• Voice AI agents run either a three-stage STT-LLM-TTS pipeline or a direct speech-to-speech model, and the choice trades control for latency.
• LiveKit's guidance puts the practical latency ceiling at one to two seconds before callers start talking over the agent, which is why WebRTC, not WebSockets or HTTP streaming, is the standard transport.
• Retell, ElevenLabs, Telnyx and xAI sell hosted platforms with different pricing models, from $0.05 to over $0.13 a minute, while LiveKit and Rasa Voice offer open-source or self-hosted paths.
• ElevenLabs' Guardrails 2.0 is still in Alpha, a sign that safety tooling for voice agents lags behind the conversational capability itself.
• Retell's August 24, 2026 release added native workflow orchestration and OpenAI's GPT-5.6 and GPT Realtime 2.1 models to its platform.

FAQ

What is the difference between a voice AI agent and an IVR system?

An IVR system routes callers through pre-recorded prompts and keypad menus using rule-based logic, so it cannot handle a question it wasn't scripted for. A voice AI agent uses a language model to understand open-ended speech, hold context across the call, and handle interruptions, which is why LiveKit's own comparison rates IVR's intelligence as "rule-based" against the voice agent's "AI-powered."

How much latency is acceptable for a voice AI agent?

Conversational turn-taking starts to break down above roughly one to two seconds of total round-trip latency, according to LiveKit's guidance, because callers begin talking over the agent. In practice, platforms tested by GetVoIP range from about 400-500ms (Synthflow) to 800-1,200ms (PolyAI), with Retell around 600-800ms.

Can voice AI agents run on open source infrastructure?

Yes. LiveKit publishes its agents framework as open source on GitHub, with 14,300 stars on the agents repository, and pairs it with an optional hosted cloud layer. Rasa Voice takes a similar self-hosted approach for enterprises that need to keep the ASR, TTS and deployment inside their own infrastructure rather than a vendor's cloud.

What is a voice AI agent guardrail?

A guardrail is a rule that constrains what an agent can say or do, enforced separately from the model's own instructions. ElevenLabs' Guardrails 2.0, currently in Alpha, applies rules at three points: reinforcing the system prompt during the conversation, screening user input for manipulation attempts, and independently checking each generated reply before it reaches the caller.

Developers comparing voice tools that aren't limited to call handling can browse the voice category on aiagentslisting.com.