Volume II · Shrink the Memory

One Librarian or Many

Sixty-four readers arrive at the reference desk with sixty-four questions, and the building has hired sixty-four librarians, one apiece. Each keeps a private card index of the same collection. The indexes are not identical — but they are far more alike than the questions are.

Where the redundancy is

Multi-head attention gives each of h heads three private projections: its own queries, its own keys, its own values. Lecture 2 showed what the second and third of those cost at generation time, and where the cost sits in the formula. The question this lecture asks is narrow and practical: are all h of those key and value sets actually earning their keep?

The empirical answer is that they are not, and the asymmetry is instructive. Keys and values across heads turn out to be considerably more redundant than queries. Queries are where the differentiation matters — one head asking about syntactic agreement while another asks which noun a pronoun refers to. The material being asked about is largely the same material. You can cut the number of indexes without cutting the number of questions.

One librarian for everyone

The first and most aggressive answer came from Noam Shazeer in 2019, in the same paper that framed decoding as a bandwidth problem. Multi-Query Attention keeps all h query heads and gives them a single shared key head and a single shared value head. In the formula from Lecture 2, n_kv_heads drops from h to 1 — a 32× or 64× cut in the cache, depending on the model, for no change in depth, context or precision.

Nothing else about attention changes. Each query head still computes its own scores, its own softmax and its own weighted sum; it simply reads them off a shared index. The arithmetic is nearly identical. Only the memory collapses.

The cut is not free. MQA carries a measurable quality loss relative to full multi-head attention, and models trained with it have shown training instability. One librarian for the whole building serves everyone; nobody is served especially well.

One librarian per department

Four years later, Ainslie and colleagues at Google published the obvious and excellent middle position. Grouped-Query Attention partitions the query heads into groups and gives each group its own key and value head. Typically four to eight query heads share one key–value head. MHA is the setting where every group has size one; MQA is the setting where there is a single group. GQA is everything in between.

Stated as furniture: multi-head attention gives each of h askers a private librarian. MQA staffs the desk with one librarian for everyone. GQA assigns one librarian per department — enough shared context to be efficient, enough separation that the questions from one department do not blur into another's.

The reported result is the reason the technique matters: GQA reaches nearly MHA quality at nearly MQA memory. It is not a compromise that splits the difference on both axes. It is a curve with a comfortable elbow, and the industry has spent the last few years sitting on that elbow.

The practicality that made it spread

A better architecture usually means retraining from scratch, which is why so many good ideas stay in papers. The GQA authors closed that gap. They showed you can uptrain an existing multi-head checkpoint into a grouped one: mean-pool the key and value heads within each group to initialise the shared head, then continue training for a small fraction of the original compute.

That single result changed the economics. A lab with a trained model and a serving bill did not have to choose between the two; it could convert what it already had. GQA became the default for open models released from 2023 onward, and it has stayed there.

What sharing gives up

Name the limitation precisely, because it is what motivates the next lecture. Grouping does not compress information. It discards it: the distinctions that the heads within a group would have drawn between their keys and values are gone, and no amount of later computation recovers them. It is a lossy trade, honestly made.

Worse, the loss is on a fixed schedule chosen in advance. When you pick eight groups you are asserting, before training and before seeing a single input, that the redundancy is distributed evenly across heads and that it falls in exactly those blocks. That is a guess. It is a reasonable one, and it has held up well in practice, but it is a guess made by a hyperparameter rather than a fact discovered from the data.