Volume VI · Where Position Lives

Clocks on the Tokens

Write each word of a sentence on its own card, then drop the cards on the floor. What you are looking at is exactly what self-attention, as defined in Lecture 1, is looking at: a heap. Order is not in the mechanism. It has to be put back by hand.

The heap

Permute the rows of the input and the rows of the output permute identically. Nothing else changes: not the scores, not the softmax, not the blend. This property is called permutation equivariance, and it follows directly from the definition — a dot product between a query and a key has no idea where either vector came from. The grid of Lecture 1 records who matches whom, never who came first.

Three families of answer have been tried. Add a position vector to the token embedding before the first layer, which is what the 2017 paper did. Bias the attention score by how far apart the two tokens are, which is what ALiBi does. Or rotate the query and the key themselves, which is what almost everything shipping today does. The three differ in where the positional information enters the computation, and that placement is what decides how gracefully the model behaves at lengths it never saw in training.

Rotation in, distance out

Rotary Position Embedding — RoPE, from Su et al., 2021 — takes the head dimensions two at a time. Each pair is a point in a plane, and a point in a plane can be rotated. For the token at position m, rotate every pair by an angle proportional to m. Do this to the queries and to the keys alike, inside the attention head, before the dot product is taken.

Now watch what happens when the score is computed. Rotating both vectors by m and n respectively, the two rotations compose, and the absolute angles cancel. What survives in the dot product is the difference of the positions and nothing else:

Absolute rotation goes in; relative distance comes out. That single line is the whole reason RoPE displaced everything before it. There is no learned position table to run out of, nothing added to the residual stream, and no separate parameter budget — the scheme is a fixed geometric transformation applied inside every head at every layer. A model that has learned to care about "three tokens back" has learned a rotation angle, not a lookup.

Six clocks

The frequencies θ_i fall geometrically with the dimension index. The first pairs turn quickly: a step of one token swings them a long way, so they resolve local order sharply and wrap around within a few dozen positions. The last pairs turn so slowly that thousands of tokens barely move them, so they carry coarse, long-range position and never wrap at all within the trained window.

That is the picture worth keeping. Each pair of dimensions is a clock hand turning at its own rate, and a position is not a number but the whole set of hand angles read together — fast hands for where you are in the clause, slow hands for where you are in the document.

Slowing the clocks

Train on four thousand tokens and ask for forty thousand and the model degrades badly. The reason is visible on the dial: past the trained length, the fast hands have wound into angular configurations the model has never been asked to interpret. It is not that the information is absent — it is that the reading is unfamiliar.

Position interpolation makes the unfamiliar familiar by refusing to leave the trained range: rescale incoming positions so that forty thousand tokens are laid out across the span the model already knows. Equivalently — and this is the version worth remembering — raise the base so the hands turn more slowly and every position you care about falls back inside the dial the model was trained to read. NTK-aware scaling refines the same move by treating the frequency bands unequally rather than squeezing them all alike.

YaRN (Peng et al., 2023) is the refinement that became the standard recipe. Leave the fast local frequencies alone, because they already complete many revolutions inside the trained window and need no help; interpolate only the slow ones, which are the bands that would otherwise be asked to represent angles they have never seen; and adjust the attention temperature to compensate for the entropy change that the longer context brings. A few hundred training steps then buy a very large extension. It is not a research curiosity: DeepSeek-V2 and V3, Qwen2.5 and Qwen3 at 128K, and gpt-oss at 131K all reach their advertised context this way.

The slope instead

ALiBi (Press et al., 2021) answers the same question by not embedding anything at all. Subtract a penalty proportional to the distance between the two tokens directly from the attention logit, before the softmax, with a fixed slope m that differs per head:

A head with a steep slope becomes nearsighted — anything far away is penalised into irrelevance — and a head with a shallow slope stays farsighted. The layer therefore contains a spread of ranges by construction rather than by training. Because the bias is a formula in the distance and not a table, it keeps producing sensible values at lengths never seen, which is precisely the property the field was chasing. BLOOM and MPT shipped it.

RoPE with YaRN has largely superseded it. But ALiBi remains the cleanest statement of the length-extrapolation problem, and the reason is worth stating plainly: what a model must not do at an unseen length is encounter a quantity it has no precedent for. ALiBi guarantees that by making the positional signal monotone and unbounded; YaRN achieves it by squeezing the unseen back into the seen. Two answers, one requirement.

The turn to no encoding at all

The result that reorganised this whole subject is negative. Kazemnejad et al. showed in 2023 that a causal decoder can learn position with no explicit encoding whatsoever. The causal mask itself leaks the information: the first token can attend to one position, the sixth to six, and a network that can count how many things it is allowed to look at can recover where it is. Position was never only in the embedding. It was in the shape of the mask all along.

Architectures of 2025 and 2026 exploit this by making position a layer-level choice rather than a global one. Llama 4's "iRoPE" interleaves local layers carrying RoPE with global layers carrying none. Cohere's Command A alternates sliding-window RoPE layers with global NoPE layers in a 3:1 ratio. SmolLM3 simply drops RoPE on every fourth layer. Kimi Linear goes furthest: no positional encoding in any of its full-attention layers, with position delegated entirely to the recency dynamics of the recurrence that surrounds them. And in partial RoPE only a fraction of each head's dimensions are rotated at all — including DeepSeek's decoupled RoPE channel inside MLA, which Lecture 4 introduced for an entirely different reason.

So the sentence to leave with is this. Position is no longer a single global decision made once at the bottom of the network. Local layers get sharp position because that is where word order is resolved; global layers often get none, because the mask supplies enough and the absence extrapolates perfectly; and in the linear hybrids of Volume V the recurrence is the positional encoding, whether or not the report says so.