Volume II · Listening — Automatic Speech Recognition

Learning to Listen Before Learning to Transcribe

Labelled speech — audio paired with a correct transcript — is expensive to produce and always scarce relative to the raw audio sitting unlabelled in the world. wav2vec 2.0's answer is to spend almost all of the learning before a single transcript is ever shown to the model.

CTC, in the previous lecture, still needed a transcript for every hour of audio it trained on. wav2vec 2.0 changes what the model spends its labelled data learning. Before it ever sees a transcript, the model is pretrained on nothing but raw, unlabelled waveforms, on a task it invents entirely from the audio itself: predict which piece of the signal was hidden. Only afterward, with a comparatively small amount of labelled speech, is it fine-tuned to actually output text.

Masking latents, not waveform samples

The architecture has three parts working in sequence. A convolutional feature encoder first turns the raw waveform into a shorter sequence of latent speech representations — a learned, compressed stand-in for the audio, not the samples themselves. Spans of these latents are then masked, hidden from the rest of the network, in the same broad spirit as masked-language-model pretraining in text, applied here to audio instead. A Transformer context network processes the whole sequence, masked spans included, and must predict, for each masked position, which of several candidate representations was the one actually hidden there.

A contrastive task over a learned codebook

The prediction task is contrastive rather than a direct regression: the true latent at a masked position is quantized against a learned codebook of candidate representations, with the selection made differentiable via Gumbel-softmax, and the model's job is to pick out that true quantized candidate from among distractor candidates drawn from elsewhere in the batch. The quantization codebook and the contrastive prediction task are learned jointly, in a single training run — the codebook is not built in a separate stage beforehand and then frozen for the contrastive objective to use.

What the pretraining actually buys

The result reported for wav2vec 2.0 is precise and striking: fine-tuned on only one hour of labelled data, it beats the prior state of the art trained on one hundred hours of labelled data — roughly a hundredfold reduction in how much labelled audio is needed to reach a given level of transcription quality. The explanation is not that fine-tuning is unusually efficient; it is that nearly all of the useful learning about what speech sounds like has already happened during the unlabelled pretraining stage, before the model has seen a single transcript.

Where it sits now

wav2vec 2.0 became the backbone of Meta's speech pipelines and the direct ancestor of HuBERT and data2vec, both of which extend the same masked-prediction idea with different targets. It remains widely used today, both as a frozen feature extractor and as a fine-tuning starting point, precisely in the low-resource settings this lecture's headline result describes — research and product pipelines where labelled speech data for the language or domain in question is scarce.

Set beside CTC, the previous lecture's mechanism, the contrast is worth stating plainly. CTC solves an alignment problem: given a transcript, it works out how that transcript corresponds to frames it has already been trained to classify. wav2vec 2.0 solves a data problem one step earlier: before any transcript enters the picture at all, it asks what a network can learn merely from the shape of speech itself. The two are not competitors — a wav2vec 2.0 encoder is routinely fine-tuned with a CTC head on top, so that the representations pretrained here become exactly the frame-level predictions the previous lecture's collapsing rule operates on.