Volume II · Bolting Vision Onto a Language Model

Cross-Attention from the Outside

Somewhere inside a large language model sits a fortune's worth of already-learned grammar, reasoning, and world knowledge. Retrain it from scratch every time you want it to notice a picture, and you have thrown that fortune away to solve a much smaller problem. Leave it alone instead, and thread a handful of new, small, carefully-gated layers through it.

Two components, one frozen giant

Flamingo, introduced by Alayrac, Donahue, Luc, Miech and colleagues at DeepMind in "Flamingo: a Visual Language Model for Few-Shot Learning," is built around a language model that never updates. Everything Flamingo adds sits either before it or threaded between its existing layers, and the pretrained language model's own weights are frozen throughout training. Two new pieces of machinery do all the work: a Perceiver Resampler that compresses visual features down to a small fixed size, and gated cross-attention layers inserted between the frozen language model's own self-attention blocks.

The Perceiver Resampler: any amount of vision, one fixed size out

A frozen vision encoder extracts visual features per image, or per frame if the input is video — and the number of such features can vary enormously depending on resolution and frame count. The Perceiver Resampler's job is to compress whatever arrives down to a small, fixed number of visual tokens, regardless of how many features went in. It does this with cross-attention in which a fixed set of learned latent queries — the same queries every time, independent of input size — attend to the (potentially huge) set of visual features. Whatever the resolution, whatever the number of frames, the output is always the same manageable size.

Gated cross-attention: letting vision in without breaking anything

The resampled visual tokens are handed to newly-inserted cross-attention layers threaded in between the frozen language model's own existing self-attention blocks. In these new layers the visual tokens act as keys and values, and the language model's own hidden states act as queries — letting the language model's running state attend to visual content at several points as it processes a sequence.

The detail that makes this survivable for an otherwise-frozen model is a learnable tanh-gate on each new cross-attention layer, initialised to zero. At the very start of training, a gate value of zero means the new layer contributes nothing at all — the language model behaves exactly as if the cross-attention layers were not there, because their output is multiplied by zero before it can affect anything downstream. As training proceeds the gate is free to move away from zero, and visual conditioning is allowed to ramp in gradually, rather than being thrown at a network that has never seen it before.

What a frozen backbone buys: in-context learning across modalities

Because the language model itself is entirely untouched and the new machinery only adds information rather than restructuring anything the model already knew, Flamingo can process arbitrarily interleaved sequences of images and text — supporting genuine in-context few-shot learning across multimodal examples, the same way a frozen text-only language model supports in-context learning across text examples alone. Show it a small number of (image, description) examples interleaved with text and it can generalise the pattern to a new image, without any gradient update at all.

Nothing about this design requires the visual and textual streams to merge into one representation. They stay two separate things throughout — a frozen stream of language reasoning, and a compressed, gated channel of visual information feeding into it from the outside. The next lecture keeps both giants frozen and asks whether even that much new machinery threaded through the language model is necessary.