Voice Agents

In plain words

Think of a support call where you speak instead of pressing 1 or 2. Your voice goes through five small stages: detect speech, turn it into text, let the model reply, turn that reply into audio, send it back. All of it must finish in about half a second, and if you start talking in the middle, the agent must stop immediately.

How it flows

A tiny example

frame = mic_audio()
if not is_speech(frame):
    return
text = speech_to_text(frame)
if text.confidence < 0.7:
    text = "could you repeat that?"
reply = think(text)
speak(reply)          # downstream
on_user_talks(cancel)  # upstream: stop speaking

Notice the last line: the cancel path runs backwards through the pipeline, and that is what lets the agent shut up mid-sentence.

What you will learn

• Why a voice agent is not just a chatbot with a speaker attached.
• The five stages every voice pipeline has, and what each one costs you in time.
• How barge-in works, so the agent shuts up when the user starts talking.
• Two well-known tools for this: Pipecat and LiveKit Agents.

The problem, simply

You have called a bank helpline. "Press 1 for balance, press 2 for card." Slow, but at least it never got confused.

Now think of the newer ones where you just speak. "Where is my order?" and a voice replies in a normal sentence. That is a voice agent. The user does not care about the model behind it. The user feels one thing only: the pause before the reply.

See, in a chat app a two second delay is nothing. You are reading, you are typing, you do not notice. In a phone call, two seconds of silence feels like the line got cut. People say "hello? hello?" and start again. So the whole engineering problem shifts. It is no longer "is the answer correct". It is "is the answer correct and did it start speaking within about half a second".

And in a call, users interrupt. Rahul asks something, the agent starts a long answer, Rahul says "no no, I meant the other order". A chatbot never faces this. A voice agent must stop mid-sentence, drop what it was going to say, and listen again.

The idea

Voice is a pipeline, not a function call

A text agent is basically one function: question in, answer out. A voice agent is a chain of small stages, each handing data to the next.

Learn these five names, they come up in every interview on this topic.

• VAD (voice activity detection): a small model that answers one question, "is a human speaking right now, or is this just fan noise?"
• STT (speech to text): turns the audio into words. Also gives you a confidence score.
• The model (the LLM): reads the text and decides what to say.
• TTS (text to speech): turns that reply into audio.
• Transport: the pipe that carries audio between the user's phone or browser and your server.

The unit moving through this chain is called a frame. A frame is just a small typed packet: this one is audio, this one is a transcript, this one is text, this one is a cancel command. Each stage is a processor that takes a frame and passes a frame on.

Two directions, not one

Here is the part people miss. Frames move downstream, mic to speaker, that is the obvious direction. But there is also an upstream direction, and it does not carry audio at all. It carries control: cancel, metrics, "the user just interrupted".

That upstream cancel is exactly how barge-in is handled. VAD notices the user started talking, sends a cancel frame backwards through the chain, TTS stops playing, and the audio already queued gets dropped. Without this the agent keeps happily talking over the user, and the call is ruined.

Warning

Warning: stopping TTS is not enough by itself. If audio is already buffered on the user's device, you must actively cut it, otherwise the user hears three more seconds of a reply you already cancelled.

The latency budget

Every stage costs time. Roughly, in a good setup: VAD 20 to 60ms, first partial transcript from STT 100 to 250ms, first token from the model 150 to 400ms, first audio out of TTS 100 to 200ms, and network round trip 30 to 80ms.

Add them up. A premium stack lands around 450 to 600ms end to end. Around 800 to 1200ms is what most teams actually ship, and it is usable. Cross 1500ms and users think the thing is broken.

Suppose Priya is building a delivery-status voice bot. She picks the biggest model for nicer answers, and its first token takes 900ms. Her total is now about 1.3 seconds. The answers are lovely and nobody stays on the call. A smaller, faster model for this one narrow task drops her to 700ms, and the same bot suddenly feels alive.

IMPRemember: sum your chain on paper before you write the code. Latency in voice is a design decision, not something you tune at the end.

The two tools people actually use

Pipecat is a Python framework built exactly around this frame-and-processor idea. You assemble the chain yourself, downstream and upstream, and plug in whichever STT or TTS provider you like. It supports several transports, including plain WebSocket and WebRTC. It also has a mode for structured conversations, where the call must follow fixed steps like verify, then confirm, then book.

LiveKit Agents comes at it from the network side. It is built on WebRTC, the same technology video calls use, and it also handles telephony, so a real phone number can reach your agent. It gives you two shapes of agent:

• MultimodalAgent — audio goes straight into a model that natively hears and speaks. No text in the middle. Fastest, but you cannot inspect or edit what was said.
• VoicePipelineAgent — the STT, then model, then TTS cascade. Slower, but you get the text at every step, so you can log it, filter it, or check it against your database.

That trade is the real interview question: speed versus control. Banking and healthcare pick the cascade because they must log and check every word. A casual assistant can take the direct-audio route.

Hosted platforms also exist on top of all this, giving you a managed voice stack without a WebRTC team. Fair choice when voice is a feature for you and not the product.

Build it

# A toy voice pipeline: VAD -> STT -> LLM -> TTS -> transport.
# Frames move DOWNSTREAM. A cancel frame moves UPSTREAM to stop speech (barge-in).

class Frame:
    def __init__(self, kind, data, ms=0, confidence=1.0):
        self.kind = kind          # audio / transcript / text / tts_audio / cancel
        self.data = data
        self.ms = ms              # time this stage took
        self.confidence = confidence

class Pipeline:
    def __init__(self, stages):
        self.stages = stages
        self.speaking = False
        self.spent_ms = 0

    def downstream(self, frame):
        for stage in self.stages:
            frame = stage(self, frame)
            if frame is None:
                print("  pipeline stopped early")
                return
            self.spent_ms += frame.ms
            print("  %-9s -> %-10s %-28r +%dms" % (
                stage.__name__, frame.kind, frame.data, frame.ms))

    def upstream(self, frame):
        # Control path: it does not carry audio, only commands.
        print("  UPSTREAM  <- %s (agent was speaking: %s)" % (frame.kind, self.speaking))
        self.speaking = False

def vad(p, f):                     # is somebody actually talking?
    return Frame("audio", f.data, ms=40)

def stt(p, f):                     # audio to text, with a confidence score
    heard, conf = f.data
    return Frame("transcript", heard, ms=180, confidence=conf)

def gate(p, f):                    # do not trust a shaky transcript
    if f.confidence < 0.7:
        return Frame("text", "Sorry, could you repeat that?", ms=5)
    return Frame("transcript", f.data, ms=0, confidence=f.confidence)

def llm(p, f):                     # our fake "model" - no API call
    if f.kind == "text":
        return Frame("text", f.data, ms=0)
    reply = "Your Chennai train is on time." if "train" in f.data.lower() else "Got it."
    return Frame("text", reply, ms=320)

def tts(p, f):                     # text to speech audio
    p.speaking = True
    return Frame("tts_audio", "[audio] " + f.data, ms=150)

def transport(p, f):               # send the audio to the caller
    return Frame("tts_audio", f.data, ms=60)

pipe = Pipeline([vad, stt, gate, llm, tts, transport])

print("Turn 1 - clear speech:")
pipe.downstream(Frame("audio", ("Is my train on time", 0.95)))
print("  total: %dms\n" % pipe.spent_ms)

print("Turn 2 - noisy hostel corridor:")
pipe.spent_ms = 0
pipe.downstream(Frame("audio", ("mmm rain hmm", 0.40)))
print("  total: %dms\n" % pipe.spent_ms)

print("Turn 3 - user interrupts while agent is speaking:")
pipe.upstream(Frame("cancel", None))
print("  agent speaking now:", pipe.speaking)

Look at three things in the output. Turn 1 prints a total near 750ms, so you can see how the cost piles up stage by stage. Turn 2 never reaches the model at all, because the confidence gate caught a bad transcript and asked the user to repeat. Turn 3 shows the upstream cancel flipping speaking back to False, which is barge-in in its simplest form.

Where you will see this

• Bank and telecom support lines that let you speak normally instead of pressing numbers.
• Food delivery and travel apps with a "talk to us" button for order and refund status.
• In-car assistants, where the driver cannot look at a screen and interruption is constant.
• Voice ordering at drive-throughs and quick service counters.
• Practice-interview and spoken-English apps that listen, score, and reply out loud.

Common mistakes

• No barge-in handling. The user interrupts and the agent keeps talking. On a real call this alone makes the product unusable, no matter how good the answers are.
• Trusting low-confidence transcripts. The STT heard noise, gave you garbage, and you fed it to the model as truth. The model then confidently answers a question nobody asked. Gate on confidence and ask the user to repeat.
• Ignoring the latency budget. Each stage adds 50 to 200ms. Teams pick the nicest model and the nicest voice separately, then wonder why the call feels dead.
• Testing only in a quiet room. Real calls have fans, traffic, and bad networks. VAD and STT behave very differently there.
• Cutting speech logically but not physically. You stop generating audio, but what was already sent keeps playing on the user's device. It feels like the agent is ignoring them.

If they ask in an interview

Q: Why can you not just add text-to-speech on top of a normal chatbot?

A: Because voice has a hard delay budget and interruptions. A chatbot can take two seconds; on a call that feels like a dropped line. And a chatbot never has to stop mid-sentence when the user cuts in, which a voice agent must do constantly.

Q: What is barge-in and how would you implement it?

A: Barge-in is the user speaking while the agent is still speaking. You detect it with voice activity detection, then send a cancel signal backwards through the pipeline so text-to-speech stops and any buffered audio is dropped. Then you go back to listening.

Q: Direct audio-to-audio model, or a speech-to-text plus model plus text-to-speech cascade?

A: Direct audio is faster because there is no text hop, but you cannot see or check the words. The cascade is slower, yet gives you text at each step to log, redact, or validate. Regulated domains almost always choose the cascade.

Try these

1. Add a counter that records milliseconds per stage, then print the slowest one. That is your first observability layer.
2. Change the confidence threshold from 0.7 to 0.3 and to 0.9. Notice how one setting lets garbage through and the other makes the agent ask "repeat?" all the time.
3. Add a very simple end-of-turn rule: treat the user as finished only if the transcript ends with ? or has more than four words. Then think about where that rule would fail.
4. Make barge-in real. Add a queue of audio chunks in the transport stage and make the upstream cancel empty that queue, not just set a flag.

Words, simply

Quick recap

• A voice agent is a five-stage pipeline, and every stage spends part of a very small time budget.
• Frames go downstream for audio and upstream for control, and that upstream path is what makes barge-in possible.
• Choose direct audio for speed, the speech-to-text cascade when you need to see and check every word.