Volume I · The Reverse of Noise

Learning to Walk Backward

Drop a bead of ink into a glass of still water. Within a minute it has spread into a pale, formless cloud — a process no one can reverse by stirring. And yet the physics that describes the spreading is simple enough to write down and, it turns out, simple enough to run in the other direction.

The physics that named the field

The name is not a metaphor borrowed loosely from chemistry; it is the literal source. Sohl-Dickstein, Weiss, Maheswaranathan and Ganguli's 2015 paper, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics," took the physical process of diffusion — the slow Markov chain by which an ink drop's structure dissolves into a uniform distribution across the water — and asked a question nobody had pushed on seriously before: if that forward process destroys structure in small, well-understood increments, can a model learn to undo each increment, and by chaining the undoings, rebuild structure from formlessness?

The forward direction is easy because it requires no learning at all. Each step adds a small, fixed amount of Gaussian noise; run enough steps and any starting distribution — the shape of a cat, the outline of a face — converges to pure noise, indifferent to where it began. The reverse direction is the entire difficulty of the field, compressed into one sentence: teach a network to invert one of those small steps, and only one, and let repetition do the rest.

A fixed forward process

Ho, Jain and Abbeel's 2020 paper, "Denoising Diffusion Probabilistic Models" (DDPM), turned that thermodynamic idea into a concrete, trainable model. The forward process is a Markov chain that adds a small, fixed amount of Gaussian noise at each of T steps: q(x_t | x_{t-1}). Nothing here is learned — the noise schedule is chosen in advance and never touched by gradient descent. What makes the chain tractable is a property of Gaussians: because each step composes linearly with the last, you can skip directly to any noise level in one step, without simulating the intermediate ones.

Read it as an instruction: to noise an image to level t, scale the original down by √ᾱ_t and add Gaussian noise scaled by the complementary term. Every step of the chain, however many there are, collapses into this one closed-form draw — which is also why training is cheap: sample a random t, sample the noise, and you have a training pair without ever touching steps 1 through t − 1.

A learned reverse process

The generative model is, in its entirety, a single learned conditional distribution p_θ(x_{t-1} | x_t), applied T times in a row starting from pure Gaussian noise. There is no separate "drawing" module and no separate "composition" module — the same small network, conditioned on the current noisy image and the timestep, is asked the same narrow question at every step: given this much noise, what did the previous, slightly cleaner version look like?

The objective, simplified to a regression

The DDPM paper derives its training objective from a variational lower bound on the data likelihood — the same ELBO machinery that underlies variational autoencoders. But the paper's headline empirical finding is that this bound can be reweighted into something far simpler and, in practice, better-behaved: instead of predicting the previous image directly, predict the noise that was added to produce the current one.

Take a clean image, pick a random timestep, add the corresponding amount of noise using the closed-form draw above, and ask the network to guess exactly which noise vector was added. That is the whole training loop. It is a plain regression problem — no adversarial game, no discriminator, no min-max instability — and it is the reason diffusion training turned out to be so much more forgiving to tune than the generative models that came before it.

The empirical results were concrete enough to be taken seriously immediately: an Inception Score of 9.46 and an FID of 3.17 on CIFAR-10, and sample quality on 256×256 LSUN scenes competitive with ProgressiveGAN — a generative adversarial architecture that had, until then, defined the state of the art for high-resolution image synthesis.

Where the complexity went

It is worth naming what this buys and what it costs. A GAN's difficulty lives in its training dynamics — two networks locked in an adversarial game that can collapse or diverge. A VAE's difficulty lives in a single encode-decode pass that has to capture everything about the data distribution at once. A diffusion model's training objective, by contrast, is almost embarrassingly simple: a regression against known noise. The complexity did not disappear. It moved — out of the objective and into the sampling procedure, which now needs hundreds or thousands of sequential network calls to produce one image.

That relocation is exactly the cost this track's landing page frames as the first of three — too many steps, no way to steer the process, too much space to work in. This lecture answers none of them. It only establishes the one fact everything else depends on: denoising, at a single fixed noise level, is a learnable skill.