Volume II · Bolting Vision Onto a Language Model

The Adapter Nobody Expected to Work

Flamingo threaded new cross-attention layers through a frozen language model. BLIP-2 trained a small transformer whose only job was to ask a frozen vision encoder useful questions. Both are answers to the same worry: that connecting two giants needs a clever piece of machinery in between. LLaVA's answer is one matrix multiplication.

The architecture with almost nothing in it

LLaVA — Visual Instruction Tuning, released in April 2023 and later selected as a NeurIPS 2023 oral in the top half of one percent of submissions — takes a frozen CLIP ViT-L vision encoder and reads off its penultimate-layer features. Those features are pushed through a single trainable linear layer whose only job is to land in the same dimensionality as the language model's own word-token embeddings. The projected image features are then simply concatenated into the input sequence alongside ordinary text tokens. There is no cross-attention layer threaded through the model, as in Lecture 3's Flamingo, and no resampler or Q-Former standing between the two towers, as in Lecture 4's BLIP-2.

The real contribution was never the architecture

A single linear layer is a strange thing to expect a paper's worth of results from, and the honest reason it worked has almost nothing to do with the projection itself. It is the training data. At the time LLaVA was built, there was no sizeable dataset of natural multi-turn conversations about images paired with instructions — the kind of data instruction-tuned language models had already learned to expect. So the authors built one, and the mechanism by which they built it is worth dwelling on, because it is genuinely surprising.

GPT-4 — a language-only model, with no visual input channel at all — was prompted with existing image captions and object bounding-box coordinates, described purely as text, and asked to synthesise plausible multi-turn instruction-following conversations about the image those captions and boxes described. GPT-4 never saw a single pixel of any image it was writing dialogue about. It was reasoning entirely from a textual description of a scene and inventing questions, answers, and reasoning chains a person might produce while looking at that scene. That process produced the first sizeable visual instruction-tuning dataset, and it is what a small linear projector was actually being trained against.

LLaVA-1.5: the same idea, tightened

Improved Baselines with Visual Instruction Tuning, published in October 2023, made three changes and no conceptual departures. The single linear layer became a two-layer MLP. The vision encoder was upgraded to a higher-resolution CLIP-ViT-L-336px variant. And the training mix added academic-task visual question-answering data, together with explicit response-formatting prompts that told the model how long an answer should be. None of this touches the core claim of § 5.1 — it simply removes rough edges from it.

The efficiency result is worth reporting exactly as stated, because it is the paper's headline: LLaVA-1.5 reached state-of-the-art results across eleven benchmarks while training a 13-billion-parameter model in roughly one day on a single node of eight A100 GPUs, using only 1.2 million data points. Contrast that against the compute Flamingo and BLIP-2 spent pretraining their bridging modules on hundreds of millions of image-text pairs — the adapter-simplicity bet was also, it turned out, a compute-simplicity bet.

LLaVA-NeXT and what came after it

LLaVA-NeXT, also known as LLaVA-1.6, is worth naming carefully: it was announced in a blog post in January 2024, not a peer-reviewed paper, and should be read with that distinction in mind. Its central mechanism increases effective input resolution by up to four times through multiple aspect-ratio image tiling — an input image is split into several tiles at different crops and aspect ratios, each encoded separately, which measurably improves performance on OCR-heavy and detailed-reasoning tasks where a single low-resolution encode loses fine text and small objects.

Two verified arXiv follow-ons continue this lineage under peer review. LLaVA-NeXT-Interleave extends the approach to interleaved multi-image and video inputs. LLaVA-OneVision, published in August 2024, unifies single-image, multi-image, and video understanding inside one model family built on the same linear/MLP-projector foundation this lecture opened with.

Why this lecture's title is meant literally

By early 2023 the field had built two increasingly sophisticated bridges in quick succession: Flamingo's gated cross-attention threaded through a frozen giant, and BLIP-2's Q-Former trained to ask a frozen vision encoder exactly the right questions. Both treated the connection between vision and language as the hard problem worth solving cleverly. LLaVA discovered that, once the training data was good enough to make up the difference, architectural sophistication in the bridge itself mattered far less than either prior lecture's design would suggest.

That lesson outlived the paper. The linear-or-MLP projector became the dominant adapter style for open vision-language models generally, in preference to Flamingo-style cross-attention. It is worth naming, without overstating the detail beyond what is confirmed, that the Qwen-VL family and InternVL both descend from this general adapter-style pattern rather than from threaded cross-attention layers — a lineage this track returns to directly in Lecture 7.