ai voice agent

How an AI Voice Agent Actually Works (And What Breaks It)

An AI voice agent is three systems in a latency budget: speech-to-text, a language model, and text-to-speech, wired to tools that can book and write records. Conversation feels natural under roughly 800 milliseconds round trip. Platforms like Vapi, Retell and ElevenLabs sell the stack; the hard parts are latency, interruption handling and what happens on failure.

What is actually inside an AI voice agent?

Four layers, chained, with a strict time budget across all of them. Understanding the chain is the whole game, because every quality problem you will hit lives in one specific link.

  1. Telephony. The call arrives over a SIP trunk or a provider like Twilio, which streams audio in both directions.
  2. Speech-to-text. Streaming transcription converts audio to text as the caller speaks, rather than waiting for them to finish. This layer also has to decide when they finished — the endpointing problem.
  3. The language model. Takes the transcript plus conversation state, decides what to say, and decides whether to call a tool.
  4. Text-to-speech. Converts the reply to audio, streamed out in chunks so the first word plays before the last is generated.

The tool calls in step three are what make it an agent rather than a phone menu. A voicebot reads options. An agent checks live calendar availability, creates a booking, looks up an order, or writes a record — and then talks about what it just did.

Why is latency the number that matters most?

Because silence is the one failure callers always notice. Target under roughly 800 milliseconds from the caller finishing a sentence to the agent starting to speak. Past about a second, people assume the line dropped or that they were misheard, and they start talking over it.

That budget has to cover the whole chain, and it is tighter than it looks:

StageRough budget
Endpointing (deciding the caller stopped)100 – 300 ms
Streaming transcription finalisation50 – 150 ms
Language model first token200 – 500 ms
Speech synthesis first audio chunk100 – 300 ms
Network and telephony overhead50 – 150 ms

Add the high end of each and you are already over budget. This is why production agents stream everything, start synthesising before the model finishes its sentence, and use a smaller, faster model for the conversational turns while reserving a larger one for genuinely hard reasoning.

It is also why the perceived quality of a voice agent tracks engineering choices more than model choice. A mediocre model at 600 ms beats an excellent one at 1,800 ms on every call.

Should you build it or buy a platform?

Buy the voice layer, own the logic. Vapi, Retell AI and ElevenLabs all solve the parts that are genuinely difficult and not differentiating for your business: telephony, streaming, barge-in, voice quality, call recording.

What you should not hand over is the business logic. Keep the prompts, the knowledge base, the integrations and the decision rules in something you control. We run that layer in n8n — the voice platform calls a webhook, n8n checks availability, writes to HubSpot, fires the confirmation SMS and returns a short result string for the agent to read back.

Two reasons that split matters. First, swapping voice vendors becomes a configuration change instead of a rebuild, and this market is moving fast enough that you will want that option. Second, the logs stay yours — and the transcripts are the asset, because they are what you tune against.

If you want the mechanics of building that backend layer, we walked through the basics in how to create workflows in n8n.

What does an AI voice agent cost to run?

Costs stack per minute across four meters, and the total typically lands between a few cents and around twenty cents a minute before platform fees.

  • Transcription — metered per audio minute.
  • Model tokens — input grows through the call as conversation history accumulates, so a long call costs more than twice a short one.
  • Speech synthesis — metered per character or per second, and the premium voices cost multiples of the standard ones.
  • Telephony — per-minute carrier cost, plus the number rental.

The variance is the real planning problem. McKinsey reported in September 2026 that completion costs for different agents can differ by as much as 30 times, because multi-step tasks can be accomplished in many ways and each path has a different token bill. On the phone that shows up as call length: an agent that handles a booking in 90 seconds and one that takes six minutes to fail cost wildly different amounts.

Two things worth doing before you scale: cap the maximum call duration, and log cost per call alongside outcome. Cost per minute is the number vendors quote. Cost per booked appointment is the number that decides whether this works.

What breaks first?

Turn-taking, and it is almost never the voice model. An agent that cannot be interrupted feels broken even with flawless synthesis.

The failures in rough order of how often they appear:

  • No barge-in. The caller tries to interrupt and the agent talks over them. Fixing this means cutting synthesis mid-word when speech is detected, and handling the half-spoken sentence in conversation state.
  • Bad endpointing. Too eager and it cuts people off mid-thought; too patient and every turn carries a dead second. Natural pauses in speech make this genuinely hard.
  • Hallucinated specifics. Prices, availability, policies. The guardrail is a refusal path — if it is not in the knowledge base, the agent says it will have someone confirm, and logs it.
  • Lost state on tool calls. A two-second calendar lookup with no filler leaves the caller in silence wondering if the call dropped. Say “let me check that” before the call, not after.
  • No failure path. The single most expensive omission. When the agent cannot proceed, it must transfer or capture and text back. Agents without this lose the hardest calls, which are usually the most valuable ones.

Does outbound calling work?

Much worse than inbound, and the gap is wide enough to change what you should build. A Forbes Technology Council analysis published in July 2026, drawing on millions of calls across one platform, reported an average of 4.7 attempts before a lead answers at all — with nearly three of those often landing on the first day.

Inbound works because the caller dialled you and is waiting. They have a question, they want it answered now, and a machine that answers in one ring beats a human who answers tomorrow.

Outbound inherits every structural problem of cold calling, then adds an agent that cannot tell a polite brush-off from genuine interest. If a vendor pitches outbound and inbound as the same product at the same success rate, that is the moment to ask for call-level data.

Where outbound does hold up is warm and transactional: appointment reminders, no-show follow-ups, confirming a delivery window, reactivating a customer who asked to be called back. The caller is expecting it, which is the whole difference.

How do you tell a good voice agent build from a bad one?

Listen to ten recordings and watch the clock. The things that separate a working system from a demo are all audible:

  1. Time the gaps. Count the silence after you stop speaking. Under a second, it works. Over two, nothing else will save it.
  2. Interrupt it. Talk over it mid-sentence. It should stop immediately and respond to what you said.
  3. Ask something it cannot know. It should decline and offer to have someone confirm, not improvise.
  4. Ask for a human. Time how long that takes and whether you have to repeat yourself.
  5. Check what landed. After a test booking, look at the calendar and the CRM. If you have to retype anything, the integration is not finished.

Those five tests take fifteen minutes and tell you more than any vendor deck. They are also, roughly, the acceptance criteria we hold our own builds to.

If you would rather have the stack assembled and the logic owned by you, that is what we do — AI voice agents, n8n backends and CRM wiring. The buyer-side view of the same decision, including real pricing, is in what an AI receptionist actually does.

What is an AI voice agent?

It is a phone or voice interface built from three chained models — speech recognition, a language model that decides what to say and which tool to call, and speech synthesis. What makes it an agent rather than a voicebot is the tool layer: it can check a calendar, create a booking or write to a CRM mid-conversation.

How fast does an AI voice agent need to respond?

Aim for under about 800 milliseconds from the caller finishing a sentence to the agent starting to speak. Humans read silence as confusion or a dropped line beyond roughly a second, and the perceived quality of the whole system tracks this number more closely than it tracks how good the language model is.

Should you build an AI voice agent or buy a platform?

Buy the voice layer and own the logic. Platforms solve telephony, streaming transcription, interruption handling and voice quality — all genuinely hard. Keep the business logic, integrations and prompts in something you control, so changing voice vendor is a config change rather than a rebuild.

What does it cost to run an AI voice agent?

Per-minute costs stack across three or four meters: transcription, model tokens, speech synthesis and telephony. Expect a few cents to around twenty cents a minute depending on voice quality and model choice, plus a platform fee. Cost per call varies far more than cost per minute, because call length is what you cannot predict.

Why do AI voice agents sound robotic even with good voices?

Usually it is turn-taking, not the voice. Agents that cannot be interrupted, that leave a beat too long before replying, or that restart a sentence after being cut off feel wrong even with excellent synthesis. Barge-in handling and endpointing do more for perceived quality than upgrading the voice model.

Can an AI voice agent make outbound calls?

Technically yes, and the results are much worse than inbound. Published platform data puts it at an average of 4.7 attempts before a lead picks up at all. Inbound works because the caller wants an answer now; outbound inherits every problem of cold calling plus an agent that cannot read a brush-off.

What should an AI voice agent never do?

It should not take payment details, make a commitment it cannot verify, or quote a price that is not in its knowledge base. Each of those is a cheap guardrail to add and an expensive thing to discover in production, because a confident wrong answer on the phone is harder to retract than a wrong chat reply.

Leave a Comment

Your email address will not be published. Required fields are marked *