Volume V · Voice Agents

The Hardest Part Is Knowing When to Talk

Two people are talking. One of them stops for three-quarters of a second — not because the sentence is finished, but because the next word takes a moment to find. Say nothing, and the pause survives; the sentence continues. Say something, and you have just talked over someone mid-thought. A voice agent has to make that call thousands of times an hour, with none of the context a human uses to make it look effortless.

Why this lecture has no textbook answer

Say so plainly, because the honesty is the point. Lecture 3 could cite CTC's forward-backward algorithm. Lecture 4 could cite wav2vec 2.0's contrastive pretraining objective. Lecture 8 could cite EnCodec's residual vector quantizer. Each of those is a mechanism with one paper that essentially settled the question. Turn-taking has no equivalent. What exists instead is an active, fragmented research area through 2025 and into 2026, several competing systems that each solve a piece of it differently, and a genuine product problem that every voice-agent team is still fighting by hand. Treat anything in this lecture as a snapshot of a moving argument, not a closed case.

The difficulty is not technical laziness. A pause in human speech is ambiguous by nature — it can mean "I am finished," "I am thinking," "I am about to add a clause," or "please jump in here." Humans resolve the ambiguity using prosody, gaze, breath, and a shared model of the conversation's shape, accumulated over a lifetime. A voice agent has access to the audio stream and, at best, a transcript running slightly behind it.

FireRedChat's turn-taking controller

One concrete, dated system worth naming is FireRedChat (arXiv:2509.06502, September 2025), a full-duplex voice interaction system built around an explicit turn-taking controller. Rather than leaving the end-of-turn decision to a single detector, it combines a streaming, personalised voice-activity detector — tuned per speaker rather than applied as one fixed threshold for everyone — with a separate end-of-turn classifier that judges whether a pause is a completion or a hesitation. The stated goal is to reduce false interruptions without becoming so cautious that it misses a genuine attempt to interrupt.

Notice what this buys and what it costs. Splitting "is someone speaking" from "has someone finished" lets each detector specialise, which should in principle beat a single blunt threshold. It also means two models must agree, in real time, on a decision neither one has complete information for — and any disagreement has to be resolved by some further policy the paper does not claim to have perfected.

Two 2026 papers, and what their existence tells you

Two further 2026 papers are worth naming not for their specific methods but for what their existence proves: this problem is still open. "Adaptive Turn-Taking for Real-time Multi-Party Voice Agents" (arXiv:2606.13544) extends the problem from a single conversation partner to several, which multiplies the ambiguity — now the system must decide not only whether to speak but to whom, and whether someone else's turn is next. "IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows" (arXiv:2606.19595) is a benchmark, and a benchmark built specifically to evaluate how well an agent recovers after a bad interruption decision is itself evidence that bad interruption decisions are still routine enough in 2026 to need a scored evaluation.

A benchmark appearing is a reliable signal in this literature. Nobody builds a standardised test for a problem the field has already put behind it.

Two philosophies, not one solved answer

The most useful thing this lecture can leave you with is not a method but a contrast, and it connects directly back to the previous lecture. Moshi's full-duplex, dual-audio-stream design makes an explicit turn-taking controller largely unnecessary in the first place. Because the model is continuously modelling both an input stream and an output stream at once, always listening and always (potentially) speaking, the boundary between turns is not a decision bolted onto the system from outside — it simply emerges from the two streams the model is already predicting. There is no separate "should I speak now" module because the architecture never separated speaking from listening to begin with.

FireRedChat-style systems take the opposite route: keep a more cascade-like or turn-based core, and bolt an explicit VAD-plus-end-of-turn-classifier controller on top of it as a dedicated decision layer. This is architecturally the more conservative choice — it does not require redesigning the underlying conversational model — but it means the turn-taking problem is solved, or not, entirely by how good that one added controller is.

Neither approach has won, and this lecture will not pretend otherwise. Present this to yourself, and to anyone you explain it to, as a genuinely open problem with at least two live architectural answers competing in production and in the literature — not as a solved textbook topic waiting for you to memorise the one correct method.