Volume III · Speaking — Text-to-Speech

Turning a Spectrogram Back Into Sound

Proving that a neural network could generate raw audio at all took a model that produces one sample and waits, then produces the next — sixteen thousand times a second. Making that fast enough for a real conversation took throwing the waiting away entirely.

The mental model

Proving that raw audio could be generated by a neural network at all took an autoregressive model predicting one sample at a time. Making that fast enough for real-time use took throwing the autoregression away entirely.

WaveNet — van den Oord, Dieleman, Zen and colleagues at DeepMind, published as "A Generative Model for Raw Audio" in September 2016 — was the proof of concept that the previous lecture's predicted mel-spectrogram could actually become a waveform. Its mechanism is stacked dilated causal convolutions, where the dilation doubles at each successive layer. That doubling gives the network an exponentially growing receptive field while using only linearly many layers — each predicted sample can condition on a long history of prior samples without needing any recurrence at all. The output at each step is a 256-way softmax over a μ-law-quantized amplitude value.

A receptive field that doubles

Figure 7a Each layer's dilation doubles, so the triangular receptive field behind the stack widens exponentially with depth while the number of layers grows only linearly. This is what lets a single output sample condition on a genuinely long history without recurrence.

A breakthrough that was too slow to use

State the limitation honestly, because the paper does not hide it: WaveNet proved that raw audio generation at full sample-rate resolution was achievable, but sampling one value at a time, fully autoregressively, was far too slow for real-time production use. Generating one second of 16kHz audio meant sixteen thousand sequential network evaluations. That is the kind of result that is a genuine breakthrough and simultaneously unusable in a live product, and both things are true of it at once — the paper is proof that raw-audio generation works, not a description of anything that shipped as a live vocoder in its original form.

Plate 7.1 Pick a clip length and watch WaveNet's sequential-step count and HiFi-GAN's single-pass time diverge by orders of magnitude — the exact 16,000-steps-per-second and 167.9×-real-time figures both papers report, made concrete for any duration.

Throwing the sequence away

HiFi-GAN — Kong, Kim and Bae, published at NeurIPS 2020 — solves the speed problem by abandoning autoregression entirely. A fully convolutional generator upsamples a mel-spectrogram to a full waveform in a single forward pass, with no step-by-step sampling anywhere in it. The architectural choice the paper identifies as central to its sample quality is its discriminators: multiple discriminators specifically designed to model periodic patterns in speech, split into multi-period and multi-scale discriminators, each catching a different kind of artefact a non-autoregressive generator might otherwise introduce.

Figure 7b Sequential versus parallel generation, drawn to scale with the idea rather than the arithmetic: WaveNet's many small dependent steps against HiFi-GAN's single pass over the whole waveform.

The reported result is 22.05kHz audio generated at 167.9× real-time speed on a V100 GPU, with quality comparable to autoregressive vocoders. That speed is exactly what made vocoding fast enough for real-time, production text-to-speech — a system that must wait sixteen thousand sequential steps to produce one second of audio cannot sit inside a live product, and one that produces that same second in a single pass can.

Where this actually runs today

HiFi-GAN, or close variants of it, is the standard vocoder backend behind most modern non-autoregressive TTS stacks — it appears, for instance, as the vocoder of choice inside NVIDIA NeMo and Coqui. Tacotron 2's own original paper paired its spectrogram predictions with a modified WaveNet; production systems since have overwhelmingly swapped that vocoder stage for something in the HiFi-GAN family, keeping the earlier lecture's split intact while replacing only the second half.

Two acts of one argument

Read together, these two papers are a single argument told in two acts. WaveNet establishes what is possible; HiFi-GAN establishes what is deployable. Neither claim substitutes for the other — a vocoder that sounds good but cannot run in real time is not a production vocoder, and the field's actual path from one to the other took exactly the four years separating these two publications. It is worth keeping both halves of that history in view: the field did not arrive at a fast vocoder directly, it arrived at a slow but working one first, and only then learned which part of the mechanism could be discarded without losing quality.

Read the primary source

WaveNet: A Generative Model for Raw Audio — van den Oord, Dieleman, Zen et al. (DeepMind), 2016.
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis — Kong, Kim, Bae, NeurIPS 2020. The 167.9× real-time figure is reported directly in the paper.
Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions — Shen et al., 2017. The vocoder's usual upstream partner, covered in the previous lecture.

❦