Volume IV · Giving Audio a Vocabulary

A Codebook for Sound

Every model in this series so far has treated audio as a continuous quantity to be regressed toward — a spectrogram value, a raw amplitude. This lecture gives it a vocabulary instead: a finite alphabet of discrete tokens that a language model can simply predict, the same move the sibling VLM track made for images.

Two papers arrive at essentially the same idea from different labs. SoundStream — Zeghidour, Luebs, Omran, Skoglund and Tagliasacchi at Google, 2021 — and EnCodec — Défossez, Copet, Synnaeve and Adi at Meta, October 2022 — are both end-to-end neural audio codecs. A convolutional encoder compresses audio into a sequence of continuous latent vectors at a low frame rate, and the central idea both papers share is how those latents then become discrete tokens at all: residual vector quantization.

One codebook is not enough

A single quantizer with one codebook could in principle map each latent vector to its nearest entry in a lookup table of learned codes. The problem is fidelity: a codebook large enough to represent audio well would need to be enormous, and an enormous codebook is exactly what a language model downstream does not want to predict over. Residual vector quantization solves this by chaining several small codebooks in cascade rather than using one large one.

Walk through the cascade precisely, because it is the whole idea of this lecture. Quantizer 1 finds the nearest entry, in its own codebook, to the original latent vector. The residual — the latent minus that codebook entry, whatever error is left over — is passed to quantizer 2, which quantizes that residual against its own separate codebook. The new residual left over from that step passes to quantizer 3, and so on for several stages. Summing every stage's chosen codebook entry back together reconstructs an increasingly accurate approximation of the original latent. More stages — a longer token sequence per audio frame — buys higher fidelity at a higher bitrate; fewer stages buys a smaller, cheaper token sequence at lower fidelity.

The same move, one modality later

The sibling VLM track's own vocabulary-building lecture covers VQ-VAE and VQGAN, which quantize an image against a single codebook. Residual vector quantization is the audio codec literature's answer to the same underlying need — a discrete alphabet a downstream model can predict over — arrived at through a different mechanism because a single codebook was not judged sufficient for audio fidelity at a usable bitrate.

What the fidelity numbers say

Both papers report the resulting fidelity is not a modest improvement but a substantial one. SoundStream at 3kbps beats Opus at 12kbps — a quarter of the bitrate of a well-established, widely deployed codec, judged higher in quality. EnCodec achieves roughly ten times smaller file size than MP3 at comparable quality, running in real time. Neither comparison is against a weak baseline; Opus and MP3 are themselves the product of decades of engineering effort on exactly this problem, which is what makes a learned codec beating them by that margin worth taking seriously rather than filing away as a lab curiosity.

What this lecture actually enables

It is worth being precise about why this is a genuine turning point rather than another representation choice among many. Every model earlier in this series predicts something continuous — a spectrogram frame, a raw amplitude — and pays for that with a specialised loss or a specialised architecture to make the regression tractable. Once audio is a sequence of discrete codes, generating it becomes exactly the same problem as generating text: next-token prediction over a fixed alphabet, with all of the machinery the sibling LLM track already built for that problem now available to reuse.