Volume III · Fusing From the Start

One Stream From Token Zero

Every architecture in the last two lectures kept two separately pretrained models and built a bridge between them afterward — a resampler, a Q-Former, a linear layer. This lecture's architecture has no bridge at all. Images and text share one vocabulary and one model from the very first token of training.

The architectural break

Chameleon, published by the Meta FAIR team in May 2024, makes a single decision that separates it from everything in Volume II. Images are converted into discrete tokens using a learned image tokenizer, in the VQGAN lineage this track met in Lecture 2. Those image tokens are then interleaved directly with ordinary text tokens in one sequence, and the whole sequence is modelled by a single standard autoregressive transformer, trained end-to-end. There is no separate frozen vision encoder anywhere in the architecture, no cross-attention adapter threaded through a frozen language model, and no Q-Former standing between two towers. There is one tokenizer, one vocabulary, and one transformer.

Report the training data precisely, since the scale is part of the argument: 2.9 trillion text-only tokens, 1.5 trillion tokens drawn from 1.4 billion image-text pairs, and 400 billion tokens from documents where image and text are already interleaved as they occur in the wild — roughly 4.8 trillion unique tokens in the mix. The paper trains for 2.1 epochs over that full mix, so the model actually sees about 9.2 trillion tokens over the course of pretraining — the size of the dataset and the size of the training run are two different numbers, and it is worth keeping them apart. Chameleon reports matching or exceeding Gemini Pro and GPT-4V on long-form mixed-modal generation tasks — tasks that ask the model to produce documents where text and images belong together, not merely to answer a question about a single picture.

Late fusion versus early fusion, named directly

It is worth drawing the contrast against all three prior architectures explicitly, because each one made the same underlying choice in a different costume. Flamingo, in Lecture 3, keeps the language model frozen and threads new cross-attention layers through it. BLIP-2, in Lecture 4, keeps both giants frozen and trains only a small bridge module between them. LLaVA, in Lecture 5, keeps the language model largely frozen, at least initially, and trains only a linear or MLP projector. All three are, architecturally, late fusion via an adapter: two separately-trained models, joined after the fact by whatever connecting piece each lecture built.

Chameleon has no such seam. There is one tokenizer, one vocabulary, one transformer, and one training objective from the start — early fusion. The distinction is not cosmetic. An adapter can only pass along whatever its frozen halves already know; an early-fused model can, in principle, learn representations shaped by both modalities jointly, because both modalities were present from the first gradient step.

Llama 4's claim, and what backs it

Meta's own announcement of Llama 4 states, in its official blog post, that Llama 4 — both the Scout and Maverick variants — uses native early fusion: text and vision tokens are integrated into a unified model backbone from the start of pretraining, with the model jointly pretrained on unlabelled text, image, and video data, without freezing the text parameters and without introducing separate multimodal-only parameters. Its vision encoder is MetaCLIP-based but, notably, trained jointly rather than kept frozen — a further break from the frozen-encoder pattern of Volume II.

That architectural detail should be held at arm's length precisely as far as its source allows. It comes only from Meta's own blog post; no accompanying peer-reviewed paper has been found describing Llama 4's fusion mechanism in the detail a technical report would carry. Treat it as a vendor-disclosed claim, not an independently verified one, and read the "native early fusion" language with that qualification attached.

The hinge point of this track

Chameleon is the direct conceptual ancestor of the unified-model line this track covers later in Volume IV — Show-o, Janus and Janus-Pro, and Emu3 — all of which build on the premise that images can be tokens like any other token. It is also the conceptual ancestor of Llama 4's stated fusion approach. This lecture is the hinge point of the whole track: where "bridge two pretrained models," the organising idea of every lecture before it, gives way to "train one model over everything from the start."