Volume I · Representing Sound

The Spectrogram — Trading Time for Frequency

A piano chord and a hiss of static can look identical as a waveform trace and sound nothing alike. The difference is not in when the pressure moved — it is in what frequencies were moving. Almost every model in this series is built to read that second thing instead of the first.

The mental model

Give up some precision about exactly when a sound happened, and in exchange learn what frequencies it's made of — nearly every model in the rest of this series consumes this trade, not the raw waveform.

The previous lecture left the waveform as a single long list of amplitudes with no internal structure a model can lean on. The spectrogram is the standard way out. It does not discard the signal; it re-expresses it. Instead of "how loud was the air at this instant," it answers "how much energy was present at this frequency, during this short stretch of time" — and it answers that question for many frequencies and many short stretches at once, laid out as a two-dimensional grid a conventional network can actually consume the way it would consume an image.

The pipeline, stage by stage

The transformation runs in a fixed sequence of stages, each one solving a specific problem the previous stage left behind. The waveform is first cut into short, overlapping frames — a window a few tens of milliseconds wide, slid along the signal a little at a time so consecutive frames overlap. Within each frame, a Fast Fourier Transform is computed: the Short-Time Fourier Transform, or STFT, applies the Fourier transform not to the whole signal at once but separately to each short frame, which is exactly the "short-time" the name refers to. Squaring the magnitude of that result gives the power spectrogram — energy per frequency bin, per frame.

Figure 2a Waveform to mel-spectrogram, one stage at a time. Each box is a transformed version of the one before it — nothing here is learned; it is fixed signal processing, computed identically for every input before any neural network runs.

Why mel, and not raw frequency

A raw power spectrogram already helps, but it treats every frequency bin as equally important, and the human ear does not work that way — it resolves low frequencies far more finely than high ones, which is also where speech carries most of its information. The mel filterbank applies a bank of triangular filters spaced along the perceptual mel scale rather than the raw frequency scale: densely packed at low frequencies, increasingly spread out at high ones. Passing the power spectrogram through these filters compresses frequency resolution the same way the ear itself compresses it, and the result — the mel-spectrogram — is the representation nearly every model in this series is actually built to consume.

Figure 2b Filters packed tightly where the ear — and speech — needs resolution, spread thin where it does not. The mel scale is this compression made explicit, applied identically to every frame of every input.

Thirty years of MFCCs, then a quiet reversal

The MFCC pipeline — mel filterbank, log, then DCT — was the dominant input representation for automatic speech recognition for roughly thirty years, powering the GMM-HMM systems of the 1980s and 1990s and the early DNN-HMM hybrids that followed them. The DCT's decorrelation step mattered enormously to those older statistical models, which assumed something close to independence between input dimensions and benefited from features engineered to satisfy that assumption.

Modern end-to-end models do not carry that assumption, and they do not take the DCT step at all. Whisper and wav2vec 2.0 — both covered later in this volume and track — consume the log-mel spectrogram directly, stopping one stage earlier than the classical pipeline did. A neural network does not need its input features pre-decorrelated the way a GMM does; it learns whatever transformation of the correlated log-mel bins it needs internally. The log-mel spectrogram, not the MFCC, is now the standard input across nearly all current speech models.

What the trade actually cost

It is worth returning, briefly, to what "trading time for frequency" gave up. The STFT's short window fixes a resolution trade-off that cannot be escaped by tuning parameters alone: a wider window resolves frequency more finely but blurs exactly when an event happened, and a narrower window sharpens timing at the cost of frequency resolution. Every mel-spectrogram in this series has made that choice somewhere, even where the lecture describing it does not dwell on the number.

Plate 2.1 Drag the STFT window wider and frequency resolution sharpens while time resolution blurs — the same N, computed both ways at once, showing the trade this lecture describes has no parameter-free escape.

That is also why the spectrogram is a way-station rather than a destination. It makes audio legible to a conventional network, but it is still a continuous, real-valued grid — nothing in it is a discrete unit a language model could predict one at a time. Volume IV's neural codec returns to exactly this signal and asks a different question of it: not how to make it legible, but how to give it a vocabulary.

Read the primary sources

S. B. Davis and P. Mermelstein, "Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Sentences," IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 28, no. 4, 1980 — the canonical MFCC reference; 1980s DSP theory with no arXiv id, cited here as the journal paper directly.
Hugging Face Audio Course — the standard current reference for the spectrogram pipeline as implemented today.

❦