Volume V · Voice Agents

Three Models Wearing a Trenchcoat

Ask an early voice assistant a question and, unseen, three strangers pass a note down a line: one writes down what you said, one decides what to answer, and one reads the answer aloud. Each is good at their own job. None of them ever hears your voice.

The pipeline, plainly

Strip away the polish and the earliest voice agents are three models this series has already met, wired in series. Automatic speech recognition — Whisper, from Volume II — transcribes the user's speech to text. A text-only large language model, the entire subject of this series' sibling LLM Attention track, reads that transcript and generates a text response. Text-to-speech — Tacotron 2 plus a vocoder, from Volume III — synthesises that reply as audio the user actually hears. Three models, three separate training runs, one pipeline.

The first cost: what the transcript throws away

The central teaching point of this lecture is that the cascade has two costs, and the first is information loss. Transcribing audio to text is inherently lossy for anything that is not encoded in the words themselves. Tone, emotion, prosody, pauses and timing, laughter, overlapping speech, non-verbal vocalisations — all of it is discarded at the ASR step. None of it can be recovered by anything downstream, because neither the LLM nor the TTS model ever sees the original audio at all. They only ever see its transcript.

Consider what that means for something as ordinary as a question asked twice, once calmly and once with visible distress. Both arrive at the language model as the same string of words. Nothing in the transcript distinguishes them, because nothing about tone survived the ASR step to be distinguished. Whatever warmth or urgency the reply ought to carry has to be reconstructed from text alone, or not carried at all — the information needed to tell the two apart was discarded before the LLM ever ran.

The second cost: latency that only adds up

The second cost is latency, and it follows directly from the strict sequencing in Figure 10a. Because each stage only begins once the previous stage has fully finished, the total response time is additive across three separate model calls, plus whatever serialization overhead sits between them — packaging a transcript for the LLM, packaging its reply for the TTS model. That is a fundamentally different latency profile from a single model that processes everything at once, and it is the profile every cascaded voice assistant before 2024 actually had.

Notice what this rules out and what it does not. A cascade is not incapable of being fast in absolute terms — each of the three models can individually be quick. What it cannot do is overlap: the LLM cannot start reasoning before ASR has finished emitting text, and TTS cannot start speaking before the LLM has finished its reply. The delay is structural, not an implementation detail any one of the three models could optimise away on its own.

The strongest statement of the problem

The clearest articulation of exactly this trade-off comes from the paper the next lecture covers in full: Défossez et al. at Kyutai, "Moshi: a speech-text foundation model for real-time dialogue" (arXiv:2410.00037, September 2024). Moshi's own stated motivation is this pipeline's two costs, named explicitly — the paper argues that cascaded systems lose non-textual information at the ASR step and cannot achieve genuine full-duplex or overlapping speech, precisely because of the strict sequencing in Figure 10a. That argument is exactly why Moshi introduces the joint audio-text architecture the next lecture is about.

What has to change

The way out is not to remove any of the three jobs a voice agent has to do — recognising speech, reasoning about a reply, and producing speech are all still necessary. What has to change is doing them as three separate models with a text bottleneck forced between every pair of them. The next lecture takes up exactly that alternative: one model, listening and speaking at once.