Volume I · Seeing and Reading Together

Turning a Picture Into Tokens

Cut a photograph into a grid of small square tiles and hand a language model the tiles in reading order, and it has no idea what it is looking at — a sequence of numbers is not automatically a sentence. Everything a vision-language model does downstream depends on the answer to one prior question: what, exactly, does a tile become?

The Vision Transformer: a patch is a vector

Dosovitskiy, Beyer, Kolesnikov and colleagues at Google proposed the most direct answer in "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." Chop the image into fixed-size, non-overlapping patches — 16×16 pixels is the paper's namesake choice — flatten each patch into a single long vector, and pass it through one linear projection, the patch embedding. Add a learned 1-D position embedding so the sequence retains some sense of where each patch sat in the original grid, prepend a [CLS] token to collect a whole-image summary, and feed the resulting sequence into a completely ordinary Transformer encoder. There is no convolutional inductive bias anywhere in this pipeline — no assumption that nearby pixels matter more than distant ones is built into the architecture at all.

That omission is deliberate, and the paper's central finding is that it is affordable: given enough pretraining data, a Vision Transformer matches or beats convolutional networks while using less compute to train. The inductive bias a convolution hard-codes turns out to be learnable instead, once there is enough data to learn it from.

It is worth naming precisely what kind of tokenization this is. Each patch becomes a vector drawn from a continuous space, not a symbol drawn from a fixed vocabulary. A ViT patch embedding has no notion of "patch #4,281 out of a vocabulary of 8,192" — every patch's vector is a fresh point in a continuous space, computed on the fly.

VQ-VAE: give images an actual vocabulary

Van den Oord, Vinyals and Kavukcuoglu at DeepMind took the opposite path in "Neural Discrete Representation Learning." Rather than leave the image representation continuous, VQ-VAE gives images a genuine vocabulary. An encoder produces continuous vectors as an intermediate step, exactly as before — but each one is then quantized: snapped to whichever entry is nearest to it in a learned, finite codebook. A decoder reconstructs the image from nothing but the resulting sequence of discrete codebook indices.

Snapping a continuous vector onto its nearest codebook entry is not a differentiable operation — there is no gradient through an argmin. VQ-VAE's solution is the straight-through estimator: on the forward pass, the quantized (snapped) vector is used; on the backward pass, gradients are copied straight through the quantization step as if it had been the identity function, letting the encoder keep learning despite the discrete bottleneck in the middle.

The motivation for accepting this awkwardness is precise and worth stating exactly: pairing a continuous latent with a powerful autoregressive decoder tends to cause posterior collapse — the decoder is expressive enough to reconstruct the image from its own internal state alone, and simply learns to ignore the latent entirely. Forcing the latent through a small, discrete vocabulary starves the decoder of that shortcut, and carries a second, arguably more important consequence for this whole lecture series: it produces tokens a language model can predict exactly the way it predicts words — a categorical distribution over a fixed vocabulary, nothing more exotic required.

VQGAN: the same codebook, a better reason to trust it

Esser, Rombach and Ommer at Heidelberg built directly on VQ-VAE's discrete-codebook idea in "Taming Transformers for High-Resolution Image Synthesis," but changed what the codebook is optimised for. VQGAN's encoder-decoder is trained with an added adversarial (GAN) loss and a perceptual loss, on top of ordinary reconstruction — producing a codebook of what the paper describes as context-rich visual parts, rather than codes chosen purely to minimise pixel-wise reconstruction error. A separate autoregressive Transformer is then trained to model long-range composition over the sequence of codebook indices, which is what lets the system synthesise high-resolution images without the transformer ever touching a raw pixel directly — it only ever predicts the next discrete code.

A fork the rest of the track keeps returning to

These two answers to "what does a patch become" are not two implementation details of one idea — they are a genuine architectural fork. The continuous, ViT-style patch embedding feeds nearly every adapter-based architecture the next volume of this series studies: a frozen vision tower hands continuous vectors to a small bridge module, which hands them to a frozen or lightly-adapted language model. The discrete, VQ-VAE/VQGAN-style visual token is the direct ancestor of a different family altogether — the early-fusion, "everything is a token" architectures that Volume III of this series takes up, where an image's tokens and a sentence's tokens sit in literally the same vocabulary and the same autoregressive stream.

Neither path is simply "correct." A continuous patch vector carries more information per token and needs no codebook to train, but it can never be predicted the way a word is — there is no finite set of outcomes to put a softmax over. A discrete code can be predicted exactly like a word, at the cost of a lossy quantization step and a codebook that has to be learned well enough not to throw away what matters in the image.