Volume I · Seeing and Reading Together

Two Towers, One Similarity Score

Show a small child a photograph of a dog and say the word "dog" enough times, and the child never has to be taught what a dog looks like in the abstract. They only ever had to learn that the word and the picture go together. Teach a model the same single fact, at web scale, and something odd happens: it turns out to already know how to name things.

Two towers, nothing generative

CLIP — Contrastive Language-Image Pre-training, introduced by Radford, Kim, Hallacy and colleagues at OpenAI in "Learning Transferable Visual Models From Natural Language Supervision" — is built from two encoders that never speak to each other during a forward pass. An image encoder (a ResNet or a Vision Transformer) reads a picture and produces a vector. A text encoder (a Transformer) reads a caption and produces a vector of the same size. That is the entire architecture. Nothing in CLIP ever learns to produce a caption from a picture, or a pixel from anything at all — there is no decoder, no generative head, nothing to sample from.

What the two towers are trained to do is agree. Project the image vector and the text vector into the same shared embedding space, and push the training so that a picture's vector and its true caption's vector end up close together — measured, as is standard for embedding spaces, by cosine similarity.

The training signal: right pairs, wrong pairs

CLIP is trained on batches of N (image, caption) pairs drawn from a dataset of 400 million such pairs collected from the web. Within a batch of N, there are exactly N correct pairings — image i with its own caption i — and N² − N incorrect ones, every image paired with every other image's caption. Training is a symmetric contrastive loss, InfoNCE-style: compute the cosine similarity between every image embedding and every text embedding in the batch, softmax-normalise each image's row of similarities across all N captions, and do the same for each caption's column across all N images. The loss pushes the correct pairing's similarity up and every incorrect pairing's similarity down, for both directions at once.

Nothing about this loss requires human-written class labels. It requires only that someone, somewhere on the web, wrote a sentence near a picture — which is how training reached 400 million pairs at all. The supervision is free-floating natural language, not a fixed taxonomy of categories.

Zero-shot classification falls out for free

Here is the payoff the mental model promises. Suppose you want CLIP to classify a photograph into one of a thousand candidate classes — a task it was never explicitly trained to do. There is no classifier head to attach, and none is trained. Instead: turn each class name into a template sentence, something like "a photo of a {class}", embed all thousand template sentences with the text encoder, embed the photograph once with the image encoder, and pick whichever class embedding lands nearest the image embedding in the shared space.

Nothing new is trained for this to work. The thousand-way classifier is assembled at inference time out of the same two towers that were trained purely to match captions to pictures. The shared embedding space already encodes "which words describe which images" — a classifier is just one particular way of asking it a question.

SigLIP: the same idea, a cheaper loss

CLIP's contrastive loss has an awkward property: normalising each row and column of the N × N similarity matrix requires a global view of every pairwise similarity in the batch, which in a distributed training setup means gathering every embedding from every device before any loss can be computed at all. Zhai, Mustafa, Kolesnikov and Beyer, working at Google DeepMind/Brain, proposed a fix in "Sigmoid Loss for Language Image Pre-Training": replace the batch-wide softmax with a per-pair sigmoid loss. Each (image, text) pair is scored independently as a binary matched/unmatched decision, with no requirement to normalise across the whole batch at all.

The consequence is practical rather than conceptual: SigLIP's loss decouples entirely from batch size and removes the expensive all-gather step, which improves training quality particularly at small batch sizes while still scaling cleanly to huge ones. The reported demonstration is concrete — a SigLiT model reached 84.5% zero-shot ImageNet accuracy training on just 4 TPUv4 chips over two days, a training budget CLIP's original recipe was never built to fit.

What this design assumes

CLIP's whole approach rests on an assumption worth stating plainly: that a single vector per image and a single vector per caption is enough to capture whether they belong together. That assumption holds well for "is this a photo of a dog" and holds far less well for questions that require reading fine detail out of a picture, or reasoning across several regions of it at once. The lectures that follow this one are, in part, a series of answers to what a single embedding vector cannot do.