← Contents Volume I · Lecture 1

Volume I · Representing Sound

1

What a Waveform Actually Is

Hold a microphone up to an ordinary sentence and, before any model ever sees it, the sentence has already become a list — tens of thousands of raw numbers long, one for every fraction of a millisecond, with nothing in the list itself marking where one word ends and the next begins.

The mental model

A recording is tens of thousands of raw pressure measurements a second. Almost nothing useful is learnable directly from that — every model in this series first has to decide what to turn it into instead.

A microphone does one thing: it measures air pressure, over and over, at a fixed rate. Point it at a person speaking and it does not hear "words" or "sounds" in any sense a person would recognise — it records a continuous wobble in pressure, and to store that wobble on a computer, something has to sample it: read off its value at a fixed number of instants per second and keep only those readings. For speech, that rate is commonly 16,000 samples a second or higher. One ordinary second of speech is therefore already sixteen thousand numbers — a single sentence is a sequence longer than most books' worth of text tokens, and it arrived that way before a single layer of any model has touched it.

ONE SECOND OF ORDINARY SPEECH ~0.003s ≈ 48 raw samples, this bracket alone
Figure 1a The waveform drawn schematically. The whole line is one second; the bracketed sliver is three thousandths of it, and even that sliver already contains dozens of raw sample points. Nothing about the line marks a phoneme boundary, a syllable, or a word — those structures exist only in the listener's head.

Timescales stacked on top of each other

The deeper difficulty is not merely the volume of numbers but what is buried inside them. A waveform mixes information that lives at wildly different timescales, all superimposed into the same one-dimensional sequence with no explicit marker for any of it. The period of a vowel's pitch repeats every few milliseconds. A syllable takes tens of milliseconds. A word or a short phrase takes whole seconds. All three are present at once, in the same list of amplitude values, and the list itself gives no hint of where one layer of structure ends and the next begins.

A model trying to learn directly from the raw sequence has to somehow discover all of these layers simultaneously, from a signal that treats a pitch period and a whole word identically — as just more numbers in the list. That is a genuinely different problem from the ones the sibling tracks in this series face, and it is worth being precise about exactly how it differs.

Neither a token nor a patch

Text arrives at a language model already broken into discrete tokens — the boundary problem is solved before the model ever runs, by a tokenizer built for exactly that purpose. Images arrive as a fixed spatial grid: a photograph is naturally chopped into a regular array of pixels, and a Vision Transformer patchifies that grid into fixed squares it can treat as a sequence, a step the sibling VLM track's Lecture 2 covers in full. Audio has neither luxury. It is a continuous signal with no natural "patch" and no natural "token" boundary anywhere in it — not a discreteness handed to it by its own format, and not a grid handed to it by geometry.

TEXT the cat sat … discrete tokens, already segmented IMAGE fixed spatial grid, patchified AUDIO continuous, no natural boundary
Figure 1b The three sibling tracks' native inputs, side by side. Text and images each arrive with a built-in unit to operate on. Audio arrives as neither — the problem the rest of this volume, and Volume IV later in the series, exist to solve.

The nearer route: stay continuous, change the quantity

The next lecture takes the nearer of two routes out of the problem this lecture has described, and it keeps the signal continuous rather than discretising it. Instead of modelling raw pressure over time, it models something derived from the same signal — energy across frequency, over time — and that substitution alone is enough to make audio tractable for a conventional network. The spectrogram trades some precision about exactly when a sound happened for a direct readout of what frequencies it is made of, and unlike the raw waveform, that readout comes with real internal structure: rows a network can treat as something like a spatial dimension, arranged in a way pitch periods, syllables and words all leave a legible trace on.

The farther route: give audio a vocabulary

The farther route, covered later in Volume IV, does not stop at making the signal more legible — it goes further and gives audio an actual discrete vocabulary, the same way images acquired one in the sibling VLM track. Once a waveform can be expressed as a sequence of entries from a fixed, learned codebook, generating audio becomes next-token prediction over that alphabet, exactly the computation a language model already performs over text tokens. That is a stronger claim than the spectrogram route makes, and it is bought at a different cost, which Volume IV covers in full.

Holding this framing steady is worth more than any formula in the next thirteen lectures. Whenever an audio model's input looks unfamiliar — a grid of frequency bins, a sequence of codebook indices — the right question is not "why didn't they just use the waveform," but "which of the two representational problems is this choice solving, and what did it give up to solve it." Every remaining lecture in this series is, in that sense, an answer to a question this one has only posed.

Read the primary source

❦