Volume II · The Continuous Picture

The Score Is a Compass

Stand somewhere on a foggy hillside with no map and only a barometer that reads atmospheric pressure. You cannot see the summit, but at every point underfoot you can feel which direction pressure drops — and following that direction, step after step, gets you to the top without ever seeing it. Diffusion's second mental model replaces "denoise" with exactly this kind of local, gradient-following instruction.

The mental model

Instead of learning to denoise at fixed steps, learn a vector field that always points toward higher data density — then just walk uphill.

The score function

The score of a distribution is ∇_x log p(x) — the gradient of the log-density taken with respect to the data point x itself, not with respect to any model parameter. That distinction is worth stating plainly because it is the most common early confusion: this is not a gradient you use to update weights during training. It is a vector field defined over the data space itself — at every point x, it points in the direction that most increases the likelihood of x under the data distribution.

Figure 3a At every point, the arrow shows which direction increases density. The score is defined everywhere in data space, not just near the peak — but its estimate is most reliable close to where data actually lives.

Where the score is hard to estimate

Song and Ermon's 2019 paper, "Generative Modeling by Estimating Gradients of the Data Distribution," names the central obstacle directly: real images occupy a thin, low-dimensional manifold inside an enormously higher-dimensional pixel space, and the density is near-zero almost everywhere off that manifold. A score estimator trained only on real data has essentially no signal to learn from in the vast regions surrounding the manifold — exactly the regions a sampler starting from random noise has to pass through on its way toward the data.

Their fix is to stop asking for the score of the clean data distribution alone. Instead, perturb the data at many different Gaussian noise scales, train one noise-conditional network to estimate the score at every scale jointly, and anneal the noise level down during sampling as the trajectory approaches the manifold. At large noise, the perturbed distribution fills much more of the space, so the score is well-defined and easy to estimate almost everywhere; as sampling proceeds and the noise scale shrinks, the estimate sharpens onto the manifold itself.

The bridge to denoising

Vincent's 2011 paper, "A Connection Between Score Matching and Denoising Autoencoders," supplies the theoretical link that makes Lecture 1 and this lecture the same computation viewed two ways: training a denoising autoencoder — predict the clean signal, or equivalently the noise, from a corrupted input — is mathematically equivalent to score matching against a smoothed, Parzen-window version of the data density. Predicting noise and estimating a score are not two competing ideas that happen to both work; one is a restatement of the other.

Sampling by following the compass: Langevin dynamics

Once a score estimate is in hand, generating a sample means starting from random noise and repeatedly nudging the current point in the direction the score points, with a little extra randomness injected at every step:

The Langevin dynamics update x_i+1 = x_i + (ε/2) ∇_x log p(x_i) + √ε · z, z ~ N(0, I)

The injected noise term is not a flaw to be tuned away. Without it, the update is plain gradient ascent on log-density, and gradient ascent climbs to the nearest peak and stops — collapsing every sample onto whichever mode it happened to start closest to. The noise keeps the trajectory exploring, so that repeated runs from different starting points populate the whole data distribution rather than a single mode of it.

Figure 3b Both paths start from the same random point. Langevin dynamics wanders, covering the distribution across many independent runs; pure gradient ascent walks straight to the nearest peak and stays there, run after run.

On CIFAR-10, this approach reached an Inception Score of 8.87 — state of the art at the time among likelihood-free generative models, and evidence that the score-matching framing was not merely a theoretical curiosity but a competitive way to generate images in its own right.

The teaching image worth keeping is a ball rolling downhill on the negative log-density landscape, jostled by noise so it never gets stuck — gradient descent with a random walk mixed in.

That equivalence is worth sitting with, because it resolves what could otherwise look like two separate lineages converging on generative image models by coincidence. They did not converge by coincidence. Lecture 1's forward-noising chain and this lecture's noise-scale ladder are describing the same object from two directions, and Lecture 4 makes that description literal by writing both as instances of one continuous equation.

Read the primary source

Generative Modeling by Estimating Gradients of the Data Distribution — Song, Ermon, 2019, NeurIPS. Noise-conditional score networks and annealed Langevin sampling.
A Connection Between Score Matching and Denoising Autoencoders — Vincent, 2011, Neural Computation. Proves denoising and score matching are the same computation.
What are Diffusion Models? — Lilian Weng. A clear walkthrough of score-based generative modeling.

❦