Volume II · Bolting Vision Onto a Language Model

A Learned Bridge

Two specialists who have never learned each other's language can still work together, provided someone stands between them who speaks both. That someone does not need to be as capable as either specialist — only good at listening to one and rephrasing for the other.

The Q-Former: a small transformer with one job

BLIP-2, introduced by Li, Li, Savarese and Hoi at Salesforce Research in "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models," inserts a single lightweight module between an entirely frozen image encoder and an entirely frozen large language model. That module is the Q-Former — a small, BERT-like transformer holding a fixed set of learnable query embeddings. Its entire job is to cross-attend to the frozen image encoder's output features and extract whichever visual information is most useful for language, compressing everything down into just those few output query vectors.

The same trick as last lecture, spent differently

It is worth being precise about what the Q-Former shares with Flamingo's Perceiver Resampler from the previous lecture, and where the two designs actually diverge. Both compress a large visual representation down into a small, fixed-size one via cross-attention with learned queries — that half of the idea is identical. The real architectural difference is what happens after compression. Flamingo threads new gated cross-attention layers through the entire frozen language model, letting visual tokens act as keys and values at several points deep inside it. BLIP-2 keeps the language model completely untouched — no new layers are inserted into it anywhere — and only ever trains the small bridge module itself, feeding the Q-Former's output into the language model as a short prefix of soft prompt tokens, via nothing more elaborate than a simple linear projection.

Two training stages, in the precise order they happen

BLIP-2's recipe is deliberately staged. Stage 1 bootstraps vision-language representation learning using only the frozen image encoder and the Q-Former — no language model is involved at all yet. This stage is trained with a combination of three losses: image-text contrastive, image-text matching, and image-captioning. Stage 2 bootstraps vision-to-language generative learning: the now-pretrained Q-Former's output queries are connected to a frozen large language model, and only the Q-Former plus a small linear projection are trained further, learning to produce output the language model can use as a prompt prefix.

The efficiency result

The reason this design is worth a full lecture rather than a footnote to Flamingo's is the reported comparison between the two: BLIP-2 beats the much larger Flamingo-80B on zero-shot VQAv2 while training 54 times fewer trainable parameters. A small, well-designed bridge module trained in two deliberate stages outperforms threading new layers through the entire frozen giant — evidence that the size of what you train need not track the size of the models you are connecting.

It is worth sitting with why the staged recipe earns that result rather than treating it as an incidental detail. Stage 1 forces the Q-Former's queries to become good at representation on their own terms — contrastive, matching, and captioning objectives, evaluated with no language model anywhere nearby to lean on. Only once those queries already carry something worth reading does Stage 2 ask them to speak in a form a frozen language model can consume. Skipping straight to Stage 2 would ask one small module to learn both jobs — extracting the right visual information and phrasing it for a language model — at the same time, out of the same limited trainable-parameter budget. Splitting the two turns out to matter more than making the bridge itself larger would have.

Neither Flamingo's threaded cross-attention nor BLIP-2's Q-Former prefix asks the vision encoder and the language model to share a single stream of tokens — both keep the two modalities in separate representations that only meet through a purpose-built connector. The next lecture in this volume asks what happens when that connector is stripped down to almost nothing at all: a single linear layer, and a great deal of care about the data it is trained on.