Volume II · Listening — Automatic Speech Recognition

From Sound to Text Without Alignment

An hour of recorded speech and its typed transcript agree on what was said. They do not agree on when. No one marks which audio frame corresponds to which letter, and hand-labelling that correspondence at scale is not a realistic option — so the network has to be made responsible for its own timing instead.

A speech recogniser, at its core, produces one output distribution per audio frame — and there are far more frames than there are letters in the transcript. Connectionist Temporal Classification, CTC, is the mechanism that turns that long, frame-by-frame stream of guesses into a short label sequence, without ever being told during training which frame corresponds to which letter. The trick has two parts: a special symbol added to the output alphabet, and a way of summing over every possible timing that could have produced the same transcript.

The blank token and the collapsing rule

CTC introduces a blank symbol, written ∅, into the network's output alphabet alongside the ordinary letters. At every frame the network emits a probability distribution over letters-plus-blank, producing one long raw path — far longer than the transcript itself, since most frames will be either blank or a repeat of the letter being currently spoken. A fixed collapsing rule turns any such raw path into a short label sequence: first merge consecutive repeated letters into one, then remove every blank. A path like ∅ h h ∅ i i i ∅ collapses, by that rule, to hi.

Summing over every path that agrees

Many different raw frame-level paths collapse to the exact same label sequence — the blank could have landed on any of several frames, and a letter could have been held for two frames or three. CTC does not pick one of these paths as correct. The probability it assigns to a given transcript is the sum, over every path that collapses to that transcript, of that path's probability. Computed naively this sum would be intractable — the number of paths grows exponentially — but it is computed exactly and efficiently by a forward-backward dynamic-programming algorithm, run over an expanded lattice that interleaves blank positions between the transcript's letters. The algorithm is the direct analogue of the forward-backward algorithm used in hidden Markov models.

Training and decoding are not the same computation

The two settings use the summed probability differently. Training maximises the summed probability of the correct transcript via ordinary gradient descent — the forward-backward computation gives an exact, differentiable value to push upward. Decoding, at inference time, does not need that sum: it uses best-path decoding or beam search to find a single likely raw path, then applies the same collapsing rule to turn it into a transcript. Training reasons over all paths at once; decoding commits to one.

Where CTC sits now

CTC powered Baidu's Deep Speech systems and Meta's early wav2letter, and for a period it was the primary decoding criterion for end-to-end neural speech recognition. It has since been superseded in that specific role — as the primary criterion — by attention-based encoder-decoder architectures, covered in Volume III, and by RNN-Transducer architectures in leading production systems. It remains widely used today, but as an auxiliary loss inside joint CTC/attention hybrid architectures rather than as the sole objective, and specifically for one property none of its successors match as cleanly: non-autoregressive, low-latency streaming decoding, since a CTC model predicts every frame independently rather than one token at a time conditioned on everything before it.