Volume IV · Steering the Process

A Classifier You Don't Need

A diffusion model that has only ever been shown how to remove noise has no idea what a cat looks like, in the sense that matters: it has no notion of "cat" as a target to aim for. Ask it to generate one and it will happily produce something plausible and generic. Guidance is the machinery that turns that generic skill toward a specific condition.

Borrowing a judge's opinion

Dhariwal and Nichol's 2021 paper, with the blunt title "Diffusion Models Beat GANs on Image Synthesis," introduced classifier guidance as the first working answer. The recipe has two separate pieces trained apart from each other. First, a classifier is trained to predict a class label from a noisy image at every noise level the diffusion process passes through — an ordinary image classifier will not do, because it has never seen an image half-buried in noise. Second, at every step of reverse sampling, the diffusion model's predicted mean is nudged in the direction of that classifier's gradient: whichever direction most increases the classifier's confidence in the target class, scaled by a guidance weight.

Put plainly: move the sample in whatever direction most increases the classifier's confidence that it is looking at the right thing. Run enough steps of that nudging and the sample drifts from generic toward specific. Dhariwal and Nichol reported an ImageNet 256×256 FID of 3.94 with classifier guidance, against 4.59 for the same unconditional architecture without it, and did so while beating BigGAN-deep using as few as 25 forward passes.

The elegant fix: no second network at all

Ho and Salimans' 2022 paper, "Classifier-Free Diffusion Guidance," removed the separate classifier entirely. Train one diffusion model to work both ways: conditionally, given a class or a prompt, and unconditionally, by randomly replacing the condition with a null token some fraction of the time during training. The same network learns both behaviours at once.

At sampling time, run that one model twice at every step — once with the real condition, once with the null token — and extrapolate outward from the difference between the two:

Read it as a direction and a push: ε_θ(x_t, c) − ε_θ(x_t, ∅) is exactly the direction the condition already moved the prediction, and a guidance weight w greater than one pushes further along that same direction than the model volunteered on its own.

This is why classifier-free guidance displaced classifier guidance almost entirely. There is no second network to train, no second training run to keep synchronized with the first, and the mechanism generalises to any condition a model can be trained to accept — a class label, a caption, anything expressible as a token to drop. It became the default guidance mechanism in GLIDE, DALL-E 3, Imagen and essentially every flow-matching model that followed, Stable Diffusion 3 from the previous lecture included.

Guidance answers which direction to move. It says nothing about how the condition itself — a whole sentence, not one class label — gets into the model's hands in the first place. That is where the next lecture picks up.