← Contents Volume V · Lecture 10

Volume V · Making It Affordable

10

Diffusion in a Smaller Room

A sculptor does not attack the marble block with the finishing chisel first. The composition gets roughed out in something forgiving — clay, or a maquette a tenth the size — where a mistake costs nothing, and only once the shape is settled does anyone reach for the tool that leaves permanent marks. Latent diffusion applies the same division of labour to an image.

The mental model

Don't denoise the full image, pixel by pixel, at every one of a thousand steps. Compress it first into something smaller that still means the same thing, denoise there instead, and only decompress at the very end.

Two stages, trained apart

Rombach, Blattmann, Lorenz, Esser and Ommer introduced latent diffusion in 2022 — the paper behind Stable Diffusion — and its central move is to split image generation into two stages that are trained entirely separately and answer entirely separate questions. The first stage asks: what is the smallest representation an image can be squeezed into without losing anything a viewer would notice? The second stage asks the question this whole course has spent nine lectures on: how do you denoise your way to a sample?

The first stage is a variational autoencoder, or VAE. It learns to compress an image into a much smaller latent representation — typically an eight-times downsampling in each spatial direction — and to reconstruct the original from that latent closely enough for practical use. Crucially, what gets thrown away is only imperceptible, high-frequency pixel detail: the fine grain of a texture, not the fact that there is a texture, still less what the image is a picture of. Semantic content survives compression; it is the last mile of pixel fidelity that gets sacrificed, and it is sacrificed on purpose, because that last mile is exactly what the second stage does not need in order to work.

FULL RESOLUTION COMPRESSED LATENT FULL RESOLUTION diffusion happens here encode denoise in latent space decode
Figure 10a The full-resolution grid on either side is never touched by the diffusion process. Every noising and denoising step in this whole course's machinery happens entirely inside the small grid in the middle — the large grid only exists at the very first and very last moment, for compression and decompression.

Why smaller is so much cheaper

The saving is not vague. Every one of the roughly thousand diffusion steps discussed in Volume I costs compute proportional to the spatial resolution and channel count being denoised — a U-Net pass over a large grid is expensive precisely because it is a large grid. Cut the spatial dimensions by eight in each direction and the per-step cost falls by roughly two orders of magnitude, because area scales with the square of the linear reduction. The VAE's own compression and decompression, by contrast, each run exactly once — at the very start and the very end of generation — so their cost barely registers against a thousand repeated denoising passes.

Plate 10.1 The square-law made concrete. Push s to 8 — Stable Diffusion's actual choice — and watch per-step cost drop to roughly 1⁄64, matching the lecture's "roughly two orders of magnitude." Push further and see the same formula keep paying out; image quality, not compute, is what eventually says stop.
THE LATENT DIFFUSION PIPELINE VAE encoder 8× smaller per spatial dim compressed latent diffusion happens here — this whole volume's worth of machinery VAE decoder runs once runs once runs on the order of a thousand times
Figure 10b Two lightweight boundary operations, each paid for once, bracketing the entire iterative process this course has spent nine lectures building. The relative widths of the boxes are illustrative, not to scale, but the relative cost is the whole point of the lecture.

Conditioning still lives here

Nothing about moving into a compressed space disturbs the conditioning mechanism from the previous lecture. The cross-attention that lets a text prompt steer denoising is threaded through the same U-Net — it simply now operates on latent-space feature maps rather than pixel-space ones. Compression and conditioning are separable concerns, which is exactly why they could be developed in separate lectures and combined without friction: one decides where the denoising happens, the other decides what steers it.

Return to the sculptor. Roughing out a composition in clay is not a lesser version of sculpting — it is a deliberate choice about which decisions belong at low resolution and which belong at high resolution. Semantic composition, the part that needs a thousand iterative refinements to get right, happens cheaply in the compressed space. Perceptual fine detail, the part that does not need iteration at all, is restored once, well, by a specialised decoder at the end. The division of labour is the entire idea: expensive iteration goes where iteration earns its keep, and everything else is done once.

This is, more than any single architectural trick discussed elsewhere in this course, the reason diffusion models became something you could run on a consumer GPU rather than only inside a research lab with a rack of accelerators. Stable Diffusion's practical availability — the fact that image generation moved from papers to laptops within about a year — traces back to this one design decision: never let the expensive, iterative part of the process touch full-resolution pixels at all.

Read the primary source

❦