← Contents Volume II · Lecture 4

Volume II · The Continuous Picture

4

One Equation, Many Algorithms

Two research groups can spend years building what look like separate machines — one counting discrete steps, one tuning a ladder of noise scales — and then discover both machines were tracing the same continuous curve all along, sampled at different points. That discovery is this lecture, and it is more than tidy bookkeeping: the continuous curve itself turns out to hand you a capability neither discrete version had.

The mental model

DDPM's discrete noising chain and score matching's many noise scales are two different discretisations of the exact same continuous process — and writing that process as a differential equation hands you a free, deterministic alternative to sampling.

The continuous limit

Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole's 2021 paper, "Score-Based Generative Modeling through Stochastic Differential Equations" (arXiv 2011.13456 — worth confirming directly rather than taking on faith, since the precise identifier is easy to mistype), takes the forward noising process from Lecture 1 and the many-noise-scale process from Lecture 3 and shows both are discrete approximations of a single continuous-time object: a stochastic differential equation.

The forward SDE dx = f(x, t) dt + g(t) dW

Read the two terms separately. The first, f(x,t)dt, is a deterministic drift — where the process would go with no randomness at all. The second, g(t)dW, is the diffusion term — random jitter injected continuously, scaled by g(t). Different choices of f and g reproduce different noising schedules; the discrete chains of Lectures 1 and 3 are what you get by taking small, fixed time steps through this same equation.

Two named special cases

Two choices of f and g matter enough to have names. The Variance Preserving (VP) SDE is built so that, discretised into finite steps, it recovers exactly DDPM's noising chain from Lecture 1. The Variance Exploding (VE) SDE, discretised the same way, recovers exactly the noise-conditional score matching of Lecture 3. One continuous object, two entirely different-looking training procedures, depending only on how it is sampled in time.

ONE CONTINUOUS CURVE, TWO DISCRETE SAMPLINGS DDPM steps (VP) score-matching scales (VE)
Figure 4a The same underlying curve, sampled with two different tick spacings. Neither DDPM nor score matching is discovering separate physics — each is one discretisation choice of a single continuous family.

The reverse-time SDE

Every forward SDE has a known, exact reverse-time SDE — a fact that follows from Anderson's time-reversal theorem for diffusion processes, established well before this paper and simply applied here:

The reverse-time SDE dx = [ f(x, t) − g(t)² ∇x log pt(x) ] dt + g(t) dW̄

Look at what appears inside the brackets: the score function, exactly as introduced in Lecture 3. This is the moment the two threads of this volume knot together — a network trained to estimate the score, or equivalently trained to denoise (Lecture 1's ε-prediction, which Vincent's result already showed is the same estimation task), has, without ever being told so, learned precisely the one quantity the reverse-time SDE needs to run backward.

The probability flow ODE

Alongside the stochastic reverse process, the same paper shows there exists a fully deterministic ordinary differential equation — no noise term at all — whose marginal distribution at every time t matches the stochastic process exactly. This single fact is the most consequential one in the whole volume for what follows: an ODE can be solved with off-the-shelf, well-understood numerical solvers, taking large steps with controlled error, in a way a stochastic process fundamentally cannot. A deterministic trajectory is what makes step-skipping — DDIM, distillation, rectified flow, all of Volume III — possible at all.

SAME START, SAME MARGINALS, TWO WAYS TO GET THERE noise clean sample reverse SDE — stochastic probability flow ODE — deterministic
Figure 4b The jagged path and the smooth path arrive at samples drawn from the same distribution when run many times. Only the smooth, deterministic one supports the large solver steps that later lectures use to skip most of the trajectory.

The reported results underline that none of this unification came at the cost of quality: CIFAR-10 FID of 2.20 and Inception Score of 9.89, competitive with or exceeding the discrete formulations it subsumes.

Treat this lecture as the volume's unification slide. DDPM and score matching stop looking like two separate research programmes that happen to both denoise images, and become two samples — one discrete-step, one noise-scale — drawn from a single continuous family. Volume III spends its three lectures entirely inside the deterministic half of that family this lecture just uncovered.

It is worth being precise about what "the same marginals" does and does not promise. The stochastic reverse process and the deterministic ODE agree on the distribution a sample is drawn from at every intermediate time t, run infinitely many times. They do not, and need not, agree on any single trajectory — which is exactly why the deterministic path is free to take shortcuts the stochastic one cannot, without changing what comes out the other end.

Read the primary source

❦