Volume III · Skip the Work

The Drain in the Sink

A basin is filling and the tap cannot be turned off. There is always exactly one unit of water arriving, whether or not anyone wants it. Sooner or later the plumbing grows a drain — and if you cap the drain because it looks unused, the room floods.

The tokens you are not allowed to forget

The finding arrived as an obstacle rather than a discovery. Researchers wanted a model that could stream indefinitely — a chat that never ends, a transcript that never closes — and the obvious way to build one is the rolling buffer of the previous lecture: hold the last w tokens, and as each new token arrives, drop the oldest. The window slides, the cache stays bounded, and nothing about the model needs to change.

It did not work. Quality did not degrade gently as old context fell away; it collapsed, and it collapsed at the precise moment the very first tokens of the sequence were evicted. This made no sense on any reading of what the model was doing. Those tokens were thousands of positions back, far outside any plausible dependency, and in many cases carried no meaning worth preserving — a system prompt's opening word, a stray newline, a beginning-of-sequence marker. Keep them and the model streamed happily. Drop them and the output turned to noise.

Softmax has to spend it

The explanation is in the denominator, and it is almost embarrassingly simple once seen. Softmax normalises: the weights over a row sum to one, always, by construction. There is no way for a head to say nothing here is relevant to this query. It can say that one thing is relevant and the rest are not, but it cannot decline to answer. The unit of mass has to land somewhere.

Attention heads are specialists. A head that tracks subject–verb agreement has nothing to do on a row of punctuation; a head that resolves pronouns is idle when there is no pronoun. For most queries, most heads have no work — and still must place their unit of attention. What they need is a position that is guaranteed to be visible, that appears in every row, and that can absorb mass without perturbing the output. Under a causal mask the first few tokens are exactly that: every query, at every position, in every layer, can see them. They become the reliable dumping ground. These are attention sinks.

Now the collapse explains itself. Delete the sinks and the mass they were absorbing does not disappear; softmax renormalises it across whatever remains. Every idle head, which had been quietly parking its unit in a corner where it did nothing, is suddenly forced to spread that unit across the very tokens it was trying to ignore. The output of the head is no longer approximately zero; it is a confident average of irrelevance. Do that in every head of every layer and the representation is corrupted from the inside.

Four tokens

StreamingLLM's fix follows directly from the diagnosis, and its economy is the reason the result is famous. Keep the rolling window, and pin the first four tokens permanently in the cache. That is the entire intervention. The cache stays O(w), the eviction policy is otherwise unchanged, no retraining is required, and the model will now generate over inputs far longer than anything it saw in training without degrading.

The technique passed quickly into production inference stacks, where the problem it solves — unbounded generation on bounded memory — is a daily operational concern rather than a research curiosity.

The anomaly was the machinery

There is a lesson here worth stating plainly, because it generalises well beyond attention. For years, anyone who plotted an attention heatmap saw the same unexplained artefact: a bright vertical stripe down the first column, present in nearly every head, corresponding to a token with no evident importance. It was noted, shrugged at, and filed under oddities of trained networks.

It was not an oddity. It was a pressure valve the model had built for itself out of the only materials available, and it was holding up the whole apparatus. Interpretability findings are sometimes descriptions of load-bearing machinery, and the fact that a phenomenon looks like a bug is not evidence that removing it is safe. The cost of learning this was a research group watching their streaming pipeline produce nonsense and having no idea why.

The sink becomes a parameter

The 2025 turn is that the workaround was promoted to a design feature. gpt-oss ships with learned per-head sink logits: an extra bias term added to the softmax denominator that corresponds to no token at all. The head can now put a share of its mass into a slot that belongs to nothing, and the weights over the real tokens sum to less than one. The model is finally able to say what it could never say before — that for this query, in this head, there is nothing worth attending to.

What was an emergent workaround, discovered by the model during training and diagnosed by researchers years later, is now a declared part of the architecture. The model no longer has to improvise a drain out of the first token in the sequence, because one was plumbed in.

The counter-move

Qwen's Gated Attention work takes the opposite view and is the more interesting for it. Rather than designing the drain in, it removes the need for one: a sigmoid gate is placed on the attention output, so a head that has nothing to contribute can multiply its own output towards zero and be done. The team reports that this stabilises training and eliminates the sink phenomenon altogether, along with the massive activations that accompany it. The approach has shipped in Qwen3-Next and Qwen3.5.

Two laboratories, the same phenomenon, opposite treatments. One builds the escape hatch into the softmax; the other gives the head a volume control after the softmax and lets the sink become unnecessary. There is no consensus here, and the disagreement is a real one about where a network's "nothing to say" should live.