A lecture series · Revised July 2026
A waveform is not a sequence of tokens, and speech is not text with a slower keyboard. Every idea in this series answers one of three problems sound raises that the sibling tracks never had to face: turning a continuous signal into something a network can predict, aligning a long audio stream to a much shorter transcript with no timing labels at all, and — the frontier as of this writing — closing the loop fast enough that a voice agent can actually hold a conversation.
Scroll slowly ↓
Speech recognition and text-to-speech were solved as two separate problems for thirty years. In 2025 they started collapsing back into one: give audio a vocabulary, the way images got one in the sibling VLM track, and a language model can generate it directly.Whisper reads text out of sound. Neural vocoders write sound out of text. Neural audio codecs made both directions the same kind of computation — next-token prediction over a discrete alphabet — and that is what makes a genuinely end-to-end voice agent possible at all, rather than three separate models standing in a trenchcoat.
A waveform is tens of thousands of samples per second of raw pressure values. Almost nothing useful is learnable directly from that.
An hour of audio and its transcript don't come with a timestamp for every word. Something has to learn the correspondence with no supervision for it at all.
A conversation cannot wait for three models to run in sequence. The gap between hearing the end of a sentence and starting to answer is where a voice agent lives or dies.