← Contents Volume III · Lecture 6

Volume III · Fewer Steps

6

Teaching the Model to Skip Itself

A staircase can be descended one tread at a time, or two at a time, or four at a time, provided you trust your footing. DDIM taught the sampler to skip treads without changing the staircase. This lecture asks a more ambitious question: can the model itself be retrained to make the whole descent in a single stride?

The mental model

Instead of changing the sampling procedure, retrain the model itself to jump further in one step — first by halving the step count over and over, then by learning to jump the whole distance in one move.

Halving the staircase

Progressive distillation, introduced by Salimans and Ho in 2022, does not try to understand the trajectory analytically the way DDIM did. It teaches a second network to imitate the first one's shortcuts. Take a fully trained diffusion model — call it the teacher — and train a fresh "student" model to match, in a single step, whatever the teacher produces in two steps. The student learns to compress a pair of small moves into one bigger one.

Once the student is trained, it becomes the new teacher, and the whole procedure repeats: a second student learns to compress two of the first student's steps into one, and so a model that once needed 8192 sampling steps needed 4096 after a single round, 2048 after two, and so on. Salimans and Ho demonstrated the distillation running all the way down to four sampling steps on CIFAR-10, ImageNet and LSUN, with limited quality loss along the way.

Plate 6.1 The halving rule turned into a counter. Drag the rounds slider and watch the step count fall exactly as the teacher–student handoff predicts — 8192, 4096, 2048, 1024 … all the way to the four-step model Salimans and Ho actually trained down to on CIFAR-10, ImageNet and LSUN.
PROGRESSIVE DISTILLATION · REPEATED HALVING start same landing EVERY STEP EVERY-OTHER EVERY-FOURTH
Figure 6a Three traces down the same staircase: the teacher taking every step, a first-round student taking every other step, and a second-round student taking every fourth step. Each round is trained only to match its immediate teacher two steps at a time — the sparser paths are a consequence, not a target set in advance.

One jump, not several halvings

Consistency models, proposed by Song, Dhariwal, Chen and Sutskever in 2023, drop the staged halving altogether and go after something more radical: a single function that maps any point along a diffusion ODE trajectory — the probability flow ODE from Lecture 4 — directly back to that trajectory's origin, the clean data point it started from.

The defining property is right there in the name: self-consistency. Two different points sampled from the same trajectory, however far apart along it, must map to the same output when passed through the function. A point near the noisy end and a point much closer to the clean end are, after all, describing the same underlying image — the function is simply being asked to agree with itself about which image that is, regardless of where along the path it is asked.

pure noise data EVERY POINT MAPS TO THE SAME ORIGIN
Figure 6b The self-consistency property drawn directly: four points along one ODE trajectory, each with its own arrow, all converging on the same clean-data endpoint. Any one of these arrows, followed on its own, is a complete one-step sample.

Because the function already knows how to land on the trajectory's origin from any point on it, genuine one-step generation becomes possible: start from pure noise, apply the function once, and read off a sample. The model can optionally be run for a few extra refinement steps when sample quality matters more than raw speed, but the one-step case is no longer a compromise — it is what the function was trained to do.

The staircase and the teleport

The two ideas in this lecture make a useful pair precisely because they disagree about what to give up. Progressive distillation is still descending the staircase — first two steps at a time, then four, then eight — it has simply gotten better at covering more ground per stride. A consistency model refuses the staircase altogether: it learns to teleport from any point on it straight to the ground floor in one move, because it was trained to already know where the ground floor is from wherever it happens to be standing.

That distinction also explains why consistency models attracted so much attention on publication. Distillation, however aggressive, is still bounded by the fact that each round only compresses what the previous round already achieved — the count of steps goes down, but a "step" remains the unit of thought. A consistency model changes the unit entirely: the object it learns is not a step at all, but a single mapping from anywhere on the trajectory to the end of it.

Further reading, held at arm's length

Two named follow-ons circulate in the 2025–26 literature — Multistep Consistency Models and Consistency Trajectory Models. Both extend the ideas here, but this lecture has not run a direct verification pass on their specific claims, so they are named here only as further reading rather than summarised as established fact.

What both halves of this lecture share with DDIM before them is the goal, not the method: fewer sequential network calls between noise and image. DDIM found a shortcut inside a fixed model. This lecture retrained the model to make the shortcut its native behaviour. The next lecture asks a different question again — not how fast a model can traverse its trajectory, but whether the trajectory itself could have been straight all along.

Read the primary source

❦