Volume V · Words as Vectors

Counting, Then Compressing

Word2vec learns by sliding a small window across the text, one local neighbourhood at a time. There is another way to reach the same kind of destination: count how often every pair of words co-occurs across the entire corpus first, then compress that one enormous table directly into vectors.

A different starting statistic

Pennington, Socher and Manning's "GloVe: Global Vectors for Word Representation" (EMNLP 2014) starts from a deliberately different place than the previous lecture. Instead of scanning local context windows the way word2vec does, GloVe first builds a global word-word co-occurrence count matrix: for every pair of words i and j, how often does word j appear in the context of word i, summed across every occurrence anywhere in the training data. Word2vec never builds this table explicitly — it never needs to, because it only ever looks at one local window at a time.

Factorizing the table

With the full co-occurrence matrix in hand, GloVe trains a log-bilinear regression model on its nonzero entries — directly factorizing global co-occurrence statistics rather than predicting from a local context window the way CBOW or skip-gram do. The word vectors that fall out of this factorization are the embeddings; nothing about a sliding window or a per-example prediction task enters the objective at all.

Two roads to a similar destination

This is the real teaching point of the two lectures taken together: word2vec and GloVe start from genuinely different statistics — local prediction across a sliding window versus global counts across the entire corpus — and land on embeddings with broadly similar properties. Neither should be treated as simply an improvement on the other. They are two different roads that happen to arrive at the same kind of place, and the honest way to hold them in mind is side by side, not ranked.

The practical difference between the two shows up mainly in what each method has to hold in memory while it works. Word2vec never assembles anything larger than one context window at a time, so its memory footprint during training stays small and roughly constant regardless of corpus size. GloVe pays that cost up front instead: building the global co-occurrence matrix requires a full pass over the corpus before any factorization begins, and the matrix itself, though sparse, scales with the square of the vocabulary. Neither cost is prohibitive at the vocabulary sizes this era worked with, but the trade is real — GloVe spends memory and a preprocessing pass to gain a training objective defined over the whole corpus at once, rather than one local window at a time.

What carries forward

Whichever road produced them, the vectors that come out of this volume all share the property that mattered for the rest of this series: a word became a point in a continuous space rather than an arbitrary symbol. That property is what every later embedding on this site inherits, whether it belongs to a word, a subword token, or — in the vision-language track's case — an image patch. The counting-versus-predicting distinction drawn in this lecture rarely survives into how later models are described, but the destination both methods reached is exactly what those later models still depend on.