A lecture series · Revised July 2026

The Great Mental Models of Seeing and Reading

A language model has no eyes. Every vision-language architecture is a different answer to the same question: how do you hand a model that only understands token sequences a picture, without either throwing away what makes it a language model or pretending vision is just another vocabulary?

Twelve lectures · Five volumes · Primary sources throughout

Scroll slowly ↓

Every VLM makes the same three decisions: how to turn pixels into something token-shaped, how much of the language model to keep frozen while that happens, and whether vision and language ever really share one stream or only ever talk through an adapter.

The field moved from bolting vision onto a frozen LLM (adapters, cross-attention, learned queries) toward fusing both modalities from the first token — and, most recently, toward models that generate images as fluently as they describe them. Each volume is a harder answer to the same three decisions.

I

The tokenizing problem

A language model consumes a sequence of discrete tokens. A photograph is neither discrete nor a sequence until something makes it one.

II

The bridging problem

Retraining a frozen, expensive LLM from scratch to understand pixels is wasteful. Every adapter architecture is a way to avoid it.

III

The generating problem

Understanding an image and producing one are different skills. Recent models try to fold both into a single objective.

Volume I

Seeing and Reading Together

2020–2023 · a shared space for two modalities

1 Two Towers, One Similarity Score CLIP trains an image encoder and a text encoder to agree on which captions match which pictures — nothing more — and zero-shot classification falls out for free. CLIP 2 Turning a Picture Into Tokens The Vision Transformer chops an image into patches and treats each one as a token. VQ-VAE and VQGAN go further and make the tokens discrete. ViT · VQ-VAE

Volume II

Bolting Vision Onto a Language Model

2022–2023 · keep the LLM frozen, add an adapter

3 Cross-Attention from the Outside Flamingo inserts new gated cross-attention layers into a frozen LM and compresses images down to a handful of tokens with a perceiver resampler. Flamingo 4 A Learned Bridge BLIP-2's Q-Former is a small transformer whose only job is to ask a frozen vision encoder useful questions and hand the answers to a frozen LLM. BLIP-2 5 The Adapter Nobody Expected to Work LLaVA replaces both of the above with a single linear layer — and GPT-4-generated instruction data turns out to matter more than adapter cleverness. LLaVA

Volume III

Fusing From the Start

2024–2026 · one stream, from token zero

6 One Stream From Token Zero Early-fusion models tokenize images the same way as text and train one autoregressive model over both from the start — no frozen half, no adapter. Early fusion 7 What 2026's Flagships Actually Do Llama 4, Gemini, and the open Qwen-VL line, compared on what each one's report actually discloses about fusing vision and language. Survey

Volume IV

Understanding and Generating

2024–2026 · one model, both directions

8 One Model, Both Directions Unified models decouple how an image is encoded for understanding from how it's encoded for generation, while sharing one transformer underneath. Janus · Show-o 9 Seeing in Time Video adds a dimension token images don't have. How frame sampling, temporal position encoding, and long video context are actually handled. Video

Volume V

Reading the Reports

how to judge what a model sees

10 Judging What a Model Sees MMMU and the other benchmarks that actually separate strong VLMs from weak ones — and the hallucination and grounding failures none of them fully catch. Evaluation 11 Pointing at the Picture Grounding and pointing as a capability, not an afterthought — what it takes for a model to say exactly where in an image it's looking. Grounding 12 Reading a VLM Report The same disclosure spectrum as the LLM and diffusion tracks, applied to multimodal labs — read end to end. Synthesis