Volume V · Voice Agents

One Model, Both Ears

Talk over someone in a real conversation and they do not wait politely for you to finish before starting to reply — hearing and speaking happen at the same time, in the same head. The previous lecture's three-model relay could never do that. This lecture is about the first models built so it stops being impossible.

The mental model

Instead of three models taking turns, one model listens and speaks on two separate audio streams at once — so interruption and overlap are native to how it works, not a feature bolted on afterward.

GPT-4o: what is disclosed, and what is not

OpenAI's "GPT-4o System Card" (arXiv:2410.21276, August 2024) describes GPT-4o as "an autoregressive omni model" that accepts and produces any mix of text, audio, image and video, "trained end-to-end across text, vision, and audio," with all modalities processed by the same network. That framing is explicitly a replacement for the three-stage pipeline the previous lecture described — one model rather than three handed off in sequence.

The latency the system card reports is precise and worth quoting exactly: an audio response as fast as 232 milliseconds, averaging 320 milliseconds — a figure the card itself compares to human conversational response time. That is a striking number set against the additive delay of Figure 10a's cascade, where three sequential model calls each contribute their own share of the wait.

What the system card does not give is any of the architectural detail that would let an outsider verify how that number is achieved. It focuses on safety evaluation, not architecture — there is no published detail on the audio tokenization scheme, the model's size, or the composition of its training data. The "true end-to-end, single network" framing is OpenAI's own characterisation of the system, and independent verification of the internal mechanism is not possible from public material. That distinction has to be stated plainly rather than smoothed over: what follows about GPT-4o is OpenAI-disclosed, not independently architecturally verified.

Moshi: the fully open counter-example

Défossez, Mazaré, Orsini et al. at Kyutai published the alternative in full: "Moshi: a speech-text foundation model for real-time dialogue" (arXiv:2410.00037, September 2024) — with code, model weights, and paper all public, at github.com/kyutai-labs/moshi. Because Moshi is fully disclosed and verifiable in a way GPT-4o's system card is not, its architecture can be described precisely rather than inferred.

Moshi is built on Helium, a 7-billion-parameter text language model pretrained on 2.1 trillion tokens. Audio is tokenized with Mimi, Kyutai's own streaming residual-vector- quantization neural codec — the same family of mechanism as the EnCodec/SoundStream codecs from Volume IV's opening lecture. On top of that foundation, Moshi models two audio streams simultaneously: one representing itself, one representing the user. A third stream runs alongside them — a time-aligned text token stream the paper calls "Inner Monologue," representing Moshi's own speech as text as it is produced. The paper reports that this text stream substantially improves linguistic quality, and that it also yields streaming ASR and streaming TTS as useful byproducts of the same architecture, rather than as separate models bolted on.

Figure 11a Two audio streams run through the same model at once — user and self — with a thin text stream tracking Moshi's own speech in parallel. Nothing here waits for anything else to finish first.

Latency is reported with the same precision the architecture allows. Theoretical latency is 160 milliseconds — 80ms from the Mimi codec's frame size plus 80ms of acoustic delay. Measured in practice on an L4 GPU, latency runs around 200 milliseconds. Every one of those figures follows from a described mechanism, not an unexplained system-level measurement.

The contrast is the lesson

Two full-duplex voice models, two different disclosure levels
Model	Latency reported	What is publicly verifiable
GPT-4o	232ms best, 320ms average	Safety evaluation only — no architecture, tokenizer, size, or data disclosed
Moshi	160ms theoretical, ~200ms measured (L4 GPU)	Full paper, weights, and code public on GitHub

Figure 11b The same architectural idea, stated by two labs — one hatched and opaque, describable only from the outside; one transparent, inspectable line by line.

Plate 11.1 Moshi's 160ms latency breaks down into two disclosed, addable pieces — codec frame size plus acoustic delay — while GPT-4o's 232–320ms is a single reported number with nothing underneath it to inspect.

The contrast is this lecture's real teaching point, and it is worth stating without softening it. Moshi is the genuinely open, independently verifiable demonstration of exactly the architectural idea GPT-4o claims for itself without disclosing. A student who wants to inspect real full-duplex mechanics in working code has exactly one place to look between the two, and it is not GPT-4o.

None of this says GPT-4o's reported numbers are false — there is no evidence here that they are. It says only that one of these two claims can be checked against a running system and the other cannot, and that difference matters for what a course on mental models can responsibly assert versus merely report.

Read the primary source

GPT-4o System Card — OpenAI, 2024. Focused on safety evaluation; architectural detail is not disclosed.
Moshi: a speech-text foundation model for real-time dialogue — Défossez et al., 2024.
Moshi code and weights — Kyutai. Fully public — the counter-example this lecture turns on.

❦