Volume IV · Steering the Process

Conditioning on Everything

Guidance, in the previous lecture, told a model which direction to move once it already had a target in mind. It never explained how the target gets there in the first place — how a whole sentence, not merely a class label out of a thousand, becomes something a denoising network can even consult. That is the gap this lecture closes.

From one label to a whole sentence

A class label is a single token drawn from a fixed, closed vocabulary — "cat," "dog," nine hundred and ninety-eight others. A caption is open-ended: any sentence in the language, describing any arrangement of any objects. Reaching that caption requires encoding it first, and the lecture before this one already established the relevant fact: Stable Diffusion 3 uses more than one pretrained text encoder together to produce that encoding. Beyond that specific, already-verified point, this lecture describes the mechanism generically, as one or more pretrained text encoders, rather than naming encoders it has not separately confirmed.

Whatever encoder produced it, a caption becomes a sequence of embeddings — one per token of the sentence, not one embedding for the sentence as a whole. That sequence has to be injected into the image-denoising network at every layer where it matters, and the mechanism that does the injecting is one this course has already spent an entire lecture on, in a different setting.

The same formula, two different modalities

Cross-attention is exactly the attention formula from Lecture 1 of the LLM track, with one change: the query comes from a different sequence than the keys and values. The image network's intermediate representations act as the query — what am I looking for, at this spatial location, right now? The text embedding sequence supplies the keys and values — what is each word about, and what does it hand over if attended to?

Q comes from the image features; K and V come from the text embeddings. This is the same equation that opened The Soft Lookup Table, the very first lecture of the LLM track — not a cousin of it, not inspired by it, the identical formula. What changed is only where Q and K, V come from: there, all three came from the same sequence attending to itself; here, they come from two different modalities, one attending to the other.

Threaded through every scale

Latent diffusion — the subject of the lecture immediately following this one — denoises a compressed representation of the image rather than the raw pixels, and it is inside that denoising network that cross-attention layers are threaded, at multiple resolutions at once. A coarse layer, working over a small spatial grid, can let the prompt influence overall layout: where the cube goes relative to the sphere. A fine layer, working over a much larger grid, can let the prompt influence detail: the exact shade the word "red" should produce.

Put the last three lectures side by side and Volume IV's whole argument comes into view. Guidance decides which direction to push a sample once a target exists. Cross-attention is what lets that target be an entire sentence rather than one label from a fixed list. In practice the two are used together — a text-conditioned model, guided by classifier-free guidance — because conditioning supplies the destination and guidance supplies the push toward it. Neither one alone was the whole story; between them, a diffusion model goes from a generic denoiser to something that will draw exactly the cube, and exactly the sphere, a sentence asked for.