Volume I · The Original Machine

The Soft Lookup Table

Attention is usually introduced as a formula. It is easier to learn — and much harder to forget — as a piece of furniture: a library where every book decides for itself how loudly to answer your question.

Three roles, one token

Every token in a sequence plays three parts at once. It asks a question — what am I looking for? It advertises itself — what am I about? And it carries a payload — what I will hand over if you choose me. Those three roles are the query, the key and the value, and they are produced by three learned projections of the same input vector.

Walk into the library with a question. You do not read every book; you glance at the spines. A spine that matches your question earns a high score, a spine that does not earns a low one. Then — and this is the part a classical lookup table cannot do — you do not take a single book off the shelf. You take a little of every book, in proportion to how well its spine matched. That mixture is the output.

The scoring is a dot product, the proportioning is a softmax, and the mixture is a weighted sum. Written out, the whole mechanism is one line:

Read it right to left and it is the library: QKᵀ scores every query against every key, the softmax turns those scores into proportions that sum to one, and multiplying by V collects the blend. Nothing in that line is learned — the learning lives entirely in the projections that produced Q, K and V.

The grid

QKᵀ is an n × n matrix: one score for every ordered pair of tokens. Drawing it is the single most useful thing you can do in this entire course, because every mechanism in the next fifteen lectures is a strategy for not filling it in.

Why divide by √d

The scaling factor looks like a detail and is not. Take two random vectors of dimension d whose entries are independent with unit variance. Their dot product has variance d, so its typical magnitude grows like √d. At d = 128 the raw scores are already spread across a range of roughly ±11.

Feed scores of that size into a softmax and it saturates: one entry takes almost all the mass and the rest collapse toward zero. A saturated softmax has a vanishing gradient, which means the projections that produced those scores stop learning. Dividing by √d_k restores unit variance, keeps the distribution soft, and keeps the gradients alive. The scale factor is not a normalisation nicety; it is what makes the layer trainable at all.

Many heads

One lookup table can only track one kind of relationship at a time. Multi-head attention runs h of them in parallel, each in its own learned subspace of dimension d/h, then concatenates the results and mixes them with a final projection. One head can follow syntactic agreement while another tracks which noun a pronoun refers to and a third simply copies the previous token.

The cost is not h times higher, because each head works in a d/h-dimensional slice. The total arithmetic is roughly the same as one full-width head; what you buy with the split is independence. That is the whole trade, and it is why h is usually a comfortable 8 to 128 rather than something carefully tuned.

What this design assumes

It is worth naming the assumptions, because later lectures break them one at a time. Attention as defined here assumes that comparing all pairs is affordable; that the softmax over the full row is necessary; that every past token deserves exact recall; and that the n × n matrix is a thing you can hold. In 2017, with 512-token sequences, all four were true.

None of them is true at a million tokens. Volume III attacks the first two, Volume IV the third, and Volume V the fourth — and Volume II attacks a fifth assumption so quiet that the original paper never states it: that each head needs its own memory.