# Voice AI agents: how phone automation works now

> How voice AI agents handle phone calls in 2026: pipeline or speech-to-speech models, latency budgets, interruptions, telephony, tool calls and consent.

- Published: 2026-10-01; last updated 2026-11-13
- Author: the WeStack AI team
- Subject: Conversational AI
- Address: https://westack.ai/blog/voice-ai-agents-guide

> A voice AI agent answers or places a phone call, listens, talks and does things in your systems while the caller is still on the line. There are two ways to build one in 2026: a pipeline of speech-to-text, a language model and text-to-speech, or a single speech-to-speech model that hears and speaks directly. The model choice matters less than latency, turn-taking, telephony and testing with real callers.

## What a voice agent is made of

Strip away the branding and every voice agent has the same parts. A phone leg that carries audio to and from the caller. Something that turns the caller's speech into meaning. Something that decides what to say and do. Something that speaks. A set of tools into your systems, such as the diary. And a way to hand the call to a person.

Take a clinic that gets 300 calls a day. Most are bookings, reschedules, opening hours and directions. That clinic is a good candidate: the calls repeat, each ends in an action a system can take, and a wrong answer about opening hours is annoying but not dangerous. A caller describing chest pain is a different matter, and the design has to send that call to a human at once.

## Pipeline or speech-to-speech?

The older design is a pipeline, sometimes called a cascade. Streaming speech-to-text produces a transcript, a language model reads it and writes a reply, and text-to-speech reads the reply out. Every stage is text you can log, inspect and filter. You can swap any part: a different transcription model for Hindi, a cheaper language model for simple intents, a voice your brand team chose.

The newer design is one model that takes audio in and produces audio out. It hears tone, hesitation and laughter that a transcript throws away, and it handles timing itself. Two we have checked at the time of writing: OpenAI's [GPT-Live-1](https://developers.openai.com/api/docs/models/gpt-live-1), released to developers in September 2026, which is full duplex (it listens while it speaks) and passes reasoning and tool use to a backend text model; and Google's Gemini 3.8 Live, [announced the same month](https://blog.google/innovation-and-ai/technology/developers-tools/build-real-time-voice-applications-gemini-audio/) for its Live API, which runs function calls in the background while it keeps talking.

That delegation detail is worth noticing. The speech model holds the conversation and a text model behind it does the thinking, so the old argument of pipeline against speech-to-speech is turning into a question of where you put the seam.

| | Pipeline | Speech-to-speech model |
|---|---|---|
| Feel of the call | Good if tuned; delays add up across three stages | Closer to a person; the model handles timing |
| Control over wording | High: text can be checked before it is spoken | Lower: you steer with instructions and test |
| Languages and voices | Widest choice; mix vendors per language | Whatever the model ships with |
| Pricing | Per component, by tokens and characters | Often per minute, plus the backend model |

Our rule of thumb: for inbound conversational calls in a language the speech models handle well, start with speech-to-speech, because the feel of the call decides whether people stay on the line. Choose a pipeline when you need a language or accent the speech models do badly, a legal script read word for word, a specific voice, or the option to run parts on your own infrastructure.

## Latency and turn-taking: why the pause matters

People are quick. Research on turn-taking puts the usual gap between one speaker finishing and the next starting at [around 200 milliseconds](https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2015.00731/full), which is less time than it takes to plan a sentence. Listeners predict the end of a turn and prepare their reply early. A machine that waits for silence and then starts thinking is already late.

On a phone call a long pause does specific damage. The caller says "hello?", the agent hears a new turn, and two replies collide. Or the caller assumes the line has dropped and hangs up.

So we write a latency budget before any code. For a pipeline, the time from the caller's last word to the agent's first sound is the sum of:

1. Endpointing: how long you wait to be sure the caller has finished.
2. Transcription finalising the last words.
3. The language model's time to first token.
4. Text-to-speech time to first audio.
5. Network and telephony transit in both directions.

Our starting target is under one second in total at the 95th percentile, measured from a recording made at the handset. That is our design target, not an industry benchmark, and the telephone network takes its share before your code runs. The ways to hit it are unglamorous: stream every stage, begin speaking on the first sentence instead of the full reply, keep first sentences short, host close to your callers, and when a tool call will take a while, say so ("let me check the diary") instead of going quiet.

Speech-to-speech models collapse stages two to four into one, which is most of their appeal, and you pay for it by the minute: OpenAI lists GPT-Live-1 at $0.05 per minute of session, with the backend model billed separately.

### Turn-taking and interruptions

Deciding when the caller has finished is harder than it sounds. A fixed silence threshold cuts people off when they pause halfway through a phone number, and waits too long after a plain "yes". Better systems add a small model that judges from the words and the sound of the speech whether the sentence is complete, on top of basic voice activity detection. LiveKit's open-source agent framework ships one.

Interruptions, which telephony people call barge-in, need three things. The agent stops talking within a fraction of a second. The conversation history is trimmed to what the caller heard, or the model will believe it delivered a sentence that was cut off. And small listening noises such as "mm-hm" are not treated as interruptions.

Full-duplex models take over much of this, with a catch that OpenAI's own guidance spells out: asking the agent to stop talking does not cancel work already running in the backend. If a caller says "no, wait, Thursday not Tuesday" while a booking is in flight, your application has to cancel or correct it.

## Telephony: numbers, SIP and transfers

A phone number comes from a carrier or a communications platform, and calls are delivered to you over SIP, the signalling protocol of internet telephony. From there you have two routes. You can terminate the call on your own media server, typically a framework such as LiveKit or Pipecat, and stream audio to whichever models you use. Or you can point the SIP trunk straight at a model provider.

OpenAI [documents the second route](https://developers.openai.com/api/docs/guides/realtime-sip). Calls go to a SIP address for your project and a webhook tells your server that one is arriving. Your server accepts or rejects it, attaches a server-side connection to follow the session and run tools, and can later transfer the call to a phone number or hang up through API calls. Signalling runs over TLS and the audio is encrypted. Google's Live API is a WebSocket service, so phone calls reach it through a telephony partner or one of the frameworks above.

Things that bite in practice:

- Phone audio is narrowband. Transcription that looks excellent on a laptop microphone gets worse on an 8 kHz phone line, so test on real call audio.
- Transfers should be warm. Pass the human a summary and the details collected, or the caller repeats everything and nobody has saved any time.
- Plan for concurrency. Providers cap simultaneous sessions, and Monday at 9 a.m. is when the clinic finds out.
- Keep keypad input as a fallback for account numbers.

## Doing things mid-call

A voice agent that only answers questions is an expensive recorded message. The value is in actions: finding the next free slot, moving an appointment, checking whether a prescription is ready. Each action is a tool call, and the [agent engineering](https://westack.ai/services/ai-agents) around those calls matters more on the phone than in chat because the caller cannot see anything.

Rules we hold to:

- Say what you are doing before a slow call, and never announce a result before the system confirms it. "You're booked" comes after the diary returns success.
- Read back the details that matter (date, time, the spelling of a name) before committing. Letters and digits are where phone audio fails most.
- Make writes idempotent, keyed to the call. Lines drop, and callers say "yes" twice.
- Verify identity before disclosing anything personal, and decide what the agent may not do by voice at all. Card numbers are our usual example: send the caller to your payment provider's own phone or link flow.
- Give every tool a timeout and a spoken fallback: offer a callback instead of a silence.

## Recording, consent and disclosure

Three separate questions get mixed together here.

**Disclosure that the caller is talking to a machine.** In the EU, [Article 50 of the AI Act](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-50) requires that people are informed when they interact with an AI system unless it is obvious, and that obligation has applied since 2 August 2026. We say it in the greeting in every market. It takes one sentence, and in our experience callers adjust how they speak once they know.

**Consent to record.** Rules differ by country and, in the United States, by state, and some require every party's agreement. Play the notice before the conversation starts and take legal advice for each market you serve.

**Outbound calling.** In February 2024 the US Federal Communications Commission [ruled](https://docs.fcc.gov/public/attachments/DOC-400393A1.pdf) that calls made with AI-generated voices count as "artificial" under the Telephone Consumer Protection Act, so the consent rules for robocalls apply to them. If your plan includes the agent ringing people, start with the lawyers, not the engineers.

Then the ordinary data questions: where audio and transcripts are stored, for how long, who can listen, and which vendors process them. A clinic's calls contain health information. Decide retention before launch, not after the first access request.

## Testing with real callers and accents

Demos use a quiet room and a confident speaker. Your callers are in cars, speak with regional accents, switch between two languages in one sentence and pause for a long time while they look for their appointment card.

What we do before launch:

1. Collect 50 to 100 real calls from the existing line, with consent, covering each task type.
2. Replay them as audio, not as typed text, so that transcription and timing are tested too.
3. Score task completion, time to first audio, wrongly heard names, dates and numbers, and what happened on each interruption.
4. Add the awkward cases on purpose: a speakerphone, background noise, long pauses, a caller who changes their mind, a caller who asks for a person.
5. Pilot on overflow or out-of-hours calls first, with a human one keypress away.

After launch, read transcripts every week and listen to the calls that ended in a hang-up within the first 15 seconds. The method is the one we describe in [how to evaluate AI agents](https://westack.ai/blog/how-to-evaluate-ai-agents), with audio added. How to word spoken replies is covered in our piece on [conversation design](https://westack.ai/blog/conversation-design-for-llm-assistants).

## Where to start

In the first week, do not pick a vendor. Listen to 50 recorded calls and sort them by intent. Choose one or two intents that are frequent, end in an action you already have an API for, and are cheap to get wrong. For the clinic that is booking and rescheduling.

The first real decision is pipeline or speech-to-speech, and language coverage and how much control you need over wording usually settle it. Then build the thinnest slice that takes a real call: one number, one intent, one tool, one transfer path, and a recording of every test call so you can measure the pause. Expect the telephony and the testing to take longer than the model work. This is the kind of system we build in our [conversational AI](https://westack.ai/services/conversational-ai) practice.

## Frequently asked questions

### How does a voice AI agent work?

The caller's audio reaches the agent over a phone line. Either a chain of three models transcribes the speech, writes a reply and speaks it, or one speech-to-speech model listens and talks in a single step. Around that sit telephony, tools into your systems and a route to a human.

### What latency should a voice agent aim for?

Aim for the caller to hear the start of a reply within about a second of finishing their sentence. People answer each other in roughly a fifth of a second, so a long silence reads as a fault and callers repeat themselves or hang up.

### Should we use a speech-to-speech model or a pipeline?

Use a speech-to-speech model when the call is conversational and the feel decides whether people stay on the line. Use a pipeline when you need a language the speech models handle badly, a script read word for word, a specific voice, or the freedom to swap any part.

### Do we have to tell callers they are talking to an AI?

In the EU, generally yes: the AI Act's transparency duty for systems that interact with people has applied since 2 August 2026. Elsewhere the rules vary, and recording a call has its own consent rules. Our advice is to say it in the greeting in every market. It costs one sentence.
