Volume V · Reading the Reports

Reading a VLM Report

Lay this track's eleven lectures side by side and a pattern was visible the entire time, quietly, without ever being named. Some of these models arrived with a paper describing exactly how vision and language were fused. Some arrived with a company blog post asserting that fusion had happened, and nothing more. This lecture names the pattern and hands you the checklist for reading the next report that lands.

The pattern, pointed at directly

Go back through this track's own evidence rather than taking the claim on faith. Lecture 7 covered Meta's Llama 4 fusion claim, and it is worth restating precisely what that claim rested on: a company blog post, with no accompanying paper describing the fusion mechanism at all. The same lecture covered Gemini's own technical report, which confirms that the model is natively multimodal while disclosing almost nothing about how — a report that answers the outcome question and is silent on the mechanism question. And the same lecture covered GPT-4o's system card, which confirms the training objective behind its multimodal design while withholding the mechanism entirely — a different silence, in the same place.

Set those three against the fully open academic line this track has relied on throughout. CLIP, ViT, Flamingo, BLIP-2, LLaVA, Chameleon, Show-o, Janus-Pro, and Emu3 are all full papers, with complete architectural description of exactly how each one turns pixels into something a language model consumes. Molmo, covered in Lecture 11, goes a step further still — not merely a paper, but an explicitly fully open release of weights, training data, and training recipe together. The contrast is not subtle once you look for it: within a single narrow research topic, multimodal fusion, disclosure varies by an enormous margin between labs, and even between one lab's own successive releases.

The checklist this track's evidence supports

Eleven lectures of evidence converge on a short, concrete list of questions worth asking of any new VLM report, in this order. What exactly turns an image into something the model consumes — continuous patches, as in Lecture 2's Vision Transformer, or a discrete vocabulary, as in the tokenizers of Lecture 2's later half? Is vision bridged into a frozen language model through an adapter, the pattern of Volume II, or fused with language from the first token of pretraining, the pattern of Volume III? What does the report actually disclose about the fusion mechanism, as opposed to merely asserting that fusion took place? And is there a real, described training recipe behind the numbers, or only a benchmark table standing in for one?

Those four questions are not new in this lecture — they are the same four decisions every volume of this track has been answering, one architecture at a time. What is new here is naming them as a single checklist and pointing out that a report's failure to answer any one of them is itself the most informative sentence in it, not an absence of information.

The arc this track actually walked

It is worth retracing the shape of the last eleven lectures as one path rather than eleven separate stops, because the shape itself is the lesson. The track opened with two towers — CLIP's separately trained image and text encoders, agreeing only on a similarity score. It moved through adapters — Flamingo's cross-attention, BLIP-2's Q-Former, LLaVA's single linear layer — each one a way of connecting two towers without retraining either from scratch. It moved through one stream, fusing vision and language from the first token rather than bridging two frozen halves, in Chameleon, Janus-Pro, and Emu3. It moved through unified understanding-and-generation in Lecture 8, video's unsettled temporal dimension in Lecture 9, and the evaluation and grounding gaps of Lectures 10 and 11. And it arrives here, at reading the report itself — which is exactly the closing move this track's sibling LLM course makes with its own final lecture.

The thesis, restated where it belongs

This track's own landing page opened with a claim worth restating exactly here, at the close: every VLM makes the same three decisions — how to tokenize pixels, how much of the language model to keep frozen while doing so, and whether vision and language ever really share one stream. Eleven lectures of primary sources were the evidence for that claim. This twelfth lecture's real lesson is the corollary the landing page did not yet spell out: a report's silence on any one of those three decisions is itself the most informative sentence in it — more informative, often, than whatever benchmark table sits beside the silence, because a benchmark table tells you how the model performed and the silence tells you what the lab was unwilling, or unable, to say about how it got there.

Read the primary sources this track has pointed to throughout, in the same spirit Lecture 11 held its own subject: provisionally, checking each claim against what the source actually discloses rather than what its press release implies. The checklist above is not a shortcut around that discipline. It is simply where to start looking.