Volume III · Speaking — Text-to-Speech

Predicting a Spectrogram From Text

Say a sentence out loud and two separate decisions have already been made before any sound leaves your mouth: what the sentence should sound like, and how to actually produce that sound with a body. Tacotron 2 splits text-to-speech along exactly that seam, and hands each half to a different specialist.

The mental model

Split text-to-speech into two separate problems — first decide what the speech should sound like, as a picture of frequencies over time, then separately turn that picture into an actual waveform.

Tacotron 2 — Shen, Pang, Weiss and colleagues at Google, published as "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions" in December 2017 — is doing, in reverse, exactly what Volume II's ASR models do. Instead of turning sound into text, it turns text into the same mel-spectrogram representation Volume I's Lecture 2 introduced, and hands the actual sound-generation problem to a separate specialist covered in the next lecture. The first stage is a recurrent sequence-to-sequence network with attention that maps character embeddings directly to mel-spectrogram frames.

Learning its own timing

Nothing in the input text says how long each word takes to say. A short function word like "the" might occupy a few frames; a stressed syllable at the end of a sentence might stretch across dozens. Tacotron 2 is never told any of this in advance. A location-sensitive attention mechanism aligns the text sequence and the acoustic frame sequence during training, letting the model discover its own correspondence between characters and the frames of audio they produce.

This is the same underlying difficulty Volume II's models face, approached from the opposite direction. There, an ASR system had to discover which stretch of audio corresponded to which letter, with no timing labels supplied. Here, the correspondence runs the other way — which stretch of predicted audio corresponds to which character — but the shape of the problem, and the reason attention rather than a fixed schedule is the tool for solving it, is identical.

Figure 6a The two-stage pipeline. Tacotron 2 itself only produces the middle box — the predicted mel-spectrogram. Turning that picture into an actual waveform is delegated entirely to a vocoder, the subject of the next lecture.

Once the mel-spectrogram is predicted, a modified WaveNet vocoder, conditioned on that spectrogram, synthesises the actual waveform. Tacotron 2 itself never touches a raw audio sample; its whole job stops at the spectrogram, and the harder problem of turning a picture of frequencies into an actual pressure wave is handed off entirely.

Figure 6b A schematic attention alignment. The near-diagonal band is the visible evidence that the model discovered its own text-to-audio timing during training — nobody supplied it with how long any word should take to say.

Handing off to a vocoder

Tacotron 2 itself never touches a raw audio sample; its whole job stops at the predicted mel-spectrogram, and the harder problem of turning a picture of frequencies into an actual pressure wave is handed off entirely. A modified WaveNet vocoder, conditioned on the predicted spectrogram, synthesises the waveform — the subject the next lecture takes up on its own terms. This is a deliberate seam, not an unfinished edge: the two halves are trained and, in later systems, swapped independently of one another, precisely because each stage is solving a different kind of problem.

The seam this lecture is really about

The interesting design decision here is not the network architecture; it is the choice to stop at a spectrogram at all. Tacotron 2 could in principle have been trained to predict raw samples directly. Splitting the problem in two — a linguistic-to-acoustic stage, then a separate acoustic-to-waveform stage — is what lets each half specialise, and it is the same split every text-to-speech system in this volume makes.

How close to human

The paper's headline result is a mean opinion score (MOS) of 4.53, against 4.58 for professionally recorded human speech — near parity with real human speech on the standard subjective quality measure used throughout speech research. That number is worth reading carefully: it is not a claim that the system is indistinguishable from a human under every condition, but that on the standard evaluation protocol, listeners rated its output almost as highly as an actual studio recording.

Where this lecture sits in the series

This lecture's place in the series is worth stating directly, because it is easy to read Tacotron 2 as simply "the TTS version of an ASR model" and move on. It is more precise than that: Volume II's models turn sound into text, and this model turns text into the very representation Volume I introduced as the standard input for almost everything in this series. Where the sound actually comes from is a question this lecture deliberately leaves open, for the next one — which is exactly why these two lectures are read as a pair rather than in isolation.

Read the primary source

Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions — Shen, Pang, Weiss et al. (Google), 2017 / ICASSP 2018. The MOS 4.53 vs 4.58 comparison is reported directly in the paper's results.

❦