Volume II · Listening — Automatic Speech Recognition

One Model, Ninety-Nine Languages

Every ASR system before this one succeeded by being careful: curated corpora, clean studio recordings, a narrow domain the model could be trusted inside. This one succeeds by being careless on purpose — by drowning any single source's quirks in 680,000 hours of whatever the internet happened to have lying around.

Whisper, published by Radford, Kim, Xu, Brockman, McLeavey and Sutskever at OpenAI in December 2022, is in one sense the least novel architecture in this entire series. It is a vanilla encoder-decoder Transformer, near-identical to the original 2017 design covered in the sibling LLM Attention track's own first lecture. There is no CTC loss anywhere in it — no forward-backward alignment algorithm at all, the mechanism that carried the whole weight of Lecture 3. There is no separate alignment stage of any kind. Text and audio are simply paired, end to end, and the network is trained directly on that pairing as an ordinary sequence-to-sequence problem: audio in, transcript out.

Everything is one sequence

What Whisper does instead of a specialised loss is fold every task the model needs to perform into the format of the sequence itself. The decoder's input is not just "start generating text" — it opens with a short run of special tokens that select, in order, the task (transcribe the audio, or translate it into English), the spoken language, and whether the output should carry timestamps at all. All of it is handled inside one single sequence-to-sequence objective. There is no separate model for translation and no separate head for language identification; the decoder simply reads a different instruction at the front of the sequence and produces a different kind of output in response.

Weak supervision, enormous scale

The training set is 680,000 hours of multilingual, multitask audio-text data, scraped from the web with minimal filtering. That description is doing real work: minimal filtering means the transcripts are frequently imperfect, the audio is frequently noisy, and the domain coverage is whatever happened to exist online rather than anything assembled to a specification. Every prior system in this volume treated data quality as the thing to protect at all costs. Whisper's premise is that at a large enough scale, and with enough diversity of source, that carefulness stops being the limiting factor — the model can learn to be robust to the mess rather than needing the mess removed first.

The paper's central verified claim follows directly from that trade: this weak-supervision-at-scale approach is zero-shot competitive with prior fully supervised systems on standard benchmarks, and approaches human-level robustness to noise and accent variation. The argument is explicit and stated plainly in the paper itself — scale and diversity of weak data beats a smaller, carefully curated, narrow-domain training set.

Why scale generalises where curation doesn't

It is worth sitting with why this generalises as well as it does, rather than simply repeating the headline number. A system trained on a narrow, clean corpus has learned exactly that corpus's assumptions about microphone quality, speaking style, and vocabulary; step outside them and accuracy falls sharply, because the model was never shown anything else. A system trained on 680,000 hours of whatever the internet actually contains has already seen the accents, the background noise, and the recording conditions that a clean corpus never included — not because anyone selected for them, but because at that scale, a web crawl simply contains them. Robustness, on this reading, is not a separate property the model was engineered to have; it is a direct consequence of what the training distribution already looked like.

Where it actually sits today

Whisper is not a research artefact that stayed in a paper. It is, at time of writing, OpenAI's own commercially served transcription API, and it is also the near-ubiquitous ASR stage inside cascaded voice-agent pipelines industry-wide, before native speech-to-speech models existed at all — the subject of Volume V's Lecture 10, which opens on exactly that cascade. It has also become the standard open-weights baseline that ASR research is measured against, in much the way a few foundational checkpoints anchor comparisons across the sibling tracks in this series. A model built to prove a point about weak supervision at scale ended up, within a couple of years, as plumbing that a large fraction of the voice-agent industry quietly depends on.