Volume III · Fewer Steps

Straightening the Path

Every method in this volume so far has treated the trajectory from noise to data as a given — a curve laid down by the diffusion process, to be walked more cleverly, or in fewer strides, or by a model retrained to jump ahead on it. This lecture asks the question none of them asked: what if the curve itself did not have to be curved?

A more general question

Flow matching, introduced by Lipman, Chen, Ben-Hamu, Nickel and Le in 2022, reframes the entire problem from the ground up. Lecture 1 derived a training objective from a diffusion process and a variational bound; Lecture 3 derived one from score matching. Flow matching skips both derivations and asks directly for a velocity field — a rule that, at every point in space and every point in time, says which direction and how fast to move a particle so that it ends up transported from noise to data.

That velocity field is trained to match a chosen conditional probability path, not a path forced on it by a fixed noising schedule. Diffusion's particular path — Gaussian, curved, dictated by the forward process — turns out to be one member of this larger family, not the only option available. Flow matching is strictly more general: it happens to contain diffusion as a special case, alongside paths that were never curved to begin with.

The simplest choice available

Rectified flow, from Liu, Gong and Liu in 2022, takes the flow-matching idea and picks the most literal path inside that larger family: pair each noise sample with a data sample, draw an actual straight line between the two, and train a network to predict the velocity along that line. Nothing about the path is learned or inferred — it is prescribed to be straight from the outset.

The payoff of prescribing straightness rather than discovering it is immediate: a single Euler step already produces a reasonable sample, because there is no curvature left for a single step to miss. Rectified flow goes further still with a procedure called reflow, which repeats the pairing-and-training process to straighten the learned paths even more with each pass.

Proof at scale: Stable Diffusion 3

Esser and colleagues demonstrated the approach at production scale in 2024 with Stable Diffusion 3. The model trains with rectified flow, with the loss reweighted toward the noise levels that matter most perceptually, and pairs that training scheme with a new architecture: the Multimodal Diffusion Transformer, or MMDiT. MMDiT keeps separate transformer weights for the text stream and the image stream, and fuses the two only inside shared attention layers — so information can flow in both directions between what the prompt says and what the image currently looks like.

The bidirectional fusion matters for a specific, longstanding weakness: rendering legible typography inside a generated image. Getting the letters in a sign or a label right requires the image pathway to keep consulting exactly which characters the prompt asked for, not merely to have been told once at the start.

Nothing here has yet asked what the model is being pointed at. A network trained only to denoise, however fast the path, produces some plausible image — not necessarily the one a prompt asked for. That is the subject the next two lectures take up: how a diffusion model, straight-path or curved, is steered toward a condition at all.