Volume I · The Reverse of Noise

What the Network Actually Predicts

Ask three photographers to describe the same darkening room and one will tell you how much light has left, one will tell you what the room still looks like, and one will give you a blend of both. All three descriptions are equivalent — you can derive any one from any other — and yet a photographer will have a strong preference depending on how dark the room already is.

One relationship, three unknowns

Recall the forward process from Lecture 1: x_t = √ᾱ_t x_0 + √(1 − ᾱ_t) ε. At any fixed timestep t, this is nothing more exotic than a linear equation relating three quantities — the clean image x₀, the noise ε, and the noisy image x_t. Given any two of the three, algebra hands you the third. That single fact is the whole content of this lecture: predicting ε, predicting x₀, and predicting any fixed linear combination of them are three ways of asking a network to output the same information.

Lecture 1's simplified objective picked one member of that family — predict ε, the noise — largely because it worked well and was easy to state. But "easy to state" is not the same as "uniformly easy to learn." The choice of target changes what the loss function penalizes hardest, and where, across the noise schedule.

Where ε-prediction struggles

ε-prediction, Lecture 1's default, works well across most of the schedule but degrades near t = 0, where very little noise has been added. There, the signal term √ᾱ_t x₀ dominates x_t almost entirely, and the noise being asked for is a small correction riding on top of something nearly clean. Extracting a small quantity from a signal that mostly isn't it is exactly the kind of numerically awkward regression that trains unevenly.

Where x0-prediction struggles

x0-prediction has the mirror-image problem. Near t = T, the noisy image carries almost no trace of the original — the network is being asked to hallucinate an entire clean image from something that looks, numerically, like pure noise. That is also awkward, but in the opposite direction: the difficulty moves from the low-noise end of the schedule to the high-noise end.

v-prediction: a target designed for uniform difficulty

v-prediction, introduced as part of the progressive-distillation line of work by Salimans and Ho, defines a weighted combination of the other two targets:

The weighting is chosen deliberately so that the prediction target has roughly uniform difficulty across the whole noise schedule, rather than concentrating difficulty at either end. It is a more numerically stable target in practice, and it becomes particularly important once sampling starts skipping steps — the subject of Volume III — where an unevenly-difficult target compounds error faster across large jumps.

Objective engineering, not architecture engineering

The lesson worth carrying forward is narrower than it first appears. Nothing about the network's architecture changes across ε-prediction, x0-prediction, or v-prediction — the same network, the same input, the same number of parameters. Only the loss function's target changes, algebraically derived from the same underlying relationship. This is a recurring move across generative modeling more broadly: the same information, reparameterized so that gradient descent has an easier time of it, rather than any new fact about the data being introduced.

Calvin Luo's unification paper is the place to see the full algebra connecting all three targets to a single reweighted objective — worth reading directly if the derivation above left the "why does this weighting work" question open, since the full proof is beyond what a lecture-length treatment can carry.

It also means a practitioner reading a paper's method section should ask, before anything else, which of the three targets it trains against — because that single choice, more than most architectural details, explains why one checkpoint samples cleanly at very few steps and another degrades the moment you push it past the schedule it was tuned on.