Volume IV · Giving Audio a Vocabulary

Cloning a Voice With a Few Seconds of Audio

Play someone three seconds of a stranger's voice and ask a system to continue speaking in it, on words that stranger never uttered. Ten lectures ago this belonged to signal processing. Once audio has a vocabulary, it belongs to exactly the same machinery that predicts the next word in a sentence.

Text-to-speech, rewritten as a sentence to finish

The previous lecture gave audio a codebook: a neural codec that turns a waveform into a short sequence of discrete tokens, the same kind of object a byte-pair tokenizer produces for text. Wang et al. at Microsoft, in "Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers" (arXiv:2301.02111, January 2023 — the paper is branded VALL-E), asked the obvious next question: if audio is now a sequence of tokens, why not generate it the way a language model generates text — by predicting the next token, conditioned on whatever came before?

The mechanism follows that idea directly. The text to be spoken is first converted to phonemes, the same unit a classical TTS front end would use. A short clip of reference audio — a few seconds recorded from whichever speaker the system is meant to sound like — is encoded into discrete codec tokens using the off-the-shelf neural codec from the previous lecture, the exact residual-vector-quantization mechanism already covered. An autoregressive Transformer, architecturally the same kind of model as a GPT-style language model, then predicts the target speaker's codec tokens, conditioned on the phonemes of the new sentence plus the reference speaker's own tokens.

One refinement keeps this fast enough to be usable. Generating every one of the codec's several quantizer layers autoregressively, one token at a time, would be slow in proportion to how many layers the codec uses. VALL-E instead predicts only the first quantizer layer autoregressively — the layer that carries the coarsest, most structurally important information — and fills in the remaining layers with a non-autoregressive model that can produce them together, in parallel.

Scale, and what a three-second prompt buys

The paper's own framing of its scale is worth reporting precisely, because it is the other half of why the approach works. VALL-E is pretrained on 60,000 hours of English speech drawn from LibriLight — a dataset the paper itself describes as "hundreds of times larger" than the training sets prior TTS systems used. That scale is what lets a single model generalise to a voice it has never encountered in training, from nothing more than a few seconds of that voice at inference time.

The result reported is zero-shot voice cloning from a three-second acoustic prompt: no fine-tuning on the target speaker, no dataset of their recordings — just three seconds, encoded into tokens, handed to the Transformer as context. The paper additionally reports that the cloned output preserves not only the prompt speaker's vocal timbre but their emotion and the acoustic environment of the recording — meaning the model is not just copying "whose voice this is" but something closer to the fuller acoustic condition the three seconds were recorded in.

What was never released

Here the honest limit of this lecture has to be stated plainly rather than papered over. Microsoft never publicly released VALL-E's model weights or its full training code. What exists in public is the paper, audio samples on a project page, and a handful of independent, smaller-scale reimplementations by other researchers — VALL-E itself remains a research demonstration, not a public product. That is a real gap in what can be verified or reproduced from outside Microsoft, and this lecture cannot close it: there is no official checkpoint to load and inspect, so the internal behaviour reported above rests on the paper's own account of it.

What it established, regardless

Whatever VALL-E's own release status, the paradigm it introduced — treating text-to-speech as codec-language-modelling, exactly the reframing this lecture opened with — did not stay confined to one unreleased system. The same lineage continued directly in VALL-E 2 and VALL-E R, both named successors built on the same core idea of predicting discrete audio tokens autoregressively from a short reference prompt. More broadly, the "TTS-as-next-token-prediction-over-codec-tokens" approach VALL-E demonstrated underlies zero-shot voice-cloning systems across the field since — the idea proved durable even where the original implementation did not become available to inspect.

That durability is the real handoff to the rest of this volume. Once speech synthesis is next-token prediction over a discrete alphabet, it can be trained, scaled and combined with other sequence-modelling tricks exactly like text — which is precisely the premise the next volume needs, when the model has to listen and speak in the same breath.