Volume V · Words as Vectors

A Word Is Defined by Its Neighbours

A linguist noticed decades ago that you can tell a great deal about a word without ever being told its definition, just by watching what tends to sit on either side of it. Turning that observation into a training objective is the whole of this lecture.

The distributional hypothesis, made computational

The linguistic idea being operationalized here is old: a word is characterized by the company it keeps. Two words that tend to appear in similar surrounding contexts probably mean similar things, even if they never appear next to each other. Mikolov, Chen, Corrado and Dean's "Efficient Estimation of Word Representations in Vector Space" (arXiv:1301.3781, 2013) turns this into a concrete prediction problem a network can be trained on directly, at a scale no earlier method had managed.

Two directions, one idea

Word2vec is not one architecture but two, and they are mirror images of each other. CBOW — continuous bag of words — predicts a word from its surrounding context words: several neighbours as input, one center word as the target. Skip-gram runs the prediction the other way: one center word as input, predicting each of its surrounding context words as the target.

Both are deliberately shallow log-linear architectures — no hidden nonlinearity of the kind the deep networks elsewhere in this series use. That simplicity is the entire point: it lets the model train on enormous corpora cheaply. The paper reports training on a corpus of 1.6 billion words in under a day.

The regularity that appears unasked

Nothing in the training objective explicitly asks for the resulting vectors to be geometrically meaningful. It only asks the model to predict context from a word, or a word from context, as well as it can across the whole corpus. But once trained, words that occurred in similar contexts land near each other in the vector space, purely as a side effect of solving that prediction problem well.

This is a useful moment to recall how the Machine Learning Foundations track defined a feature: a hand-chosen, explicit measurement of the input, built by a person who had to decide in advance what mattered. Word2vec's vectors are the opposite of that — no one specifies what any of the three hundred-odd dimensions of a word vector "mean." The training objective only asks for good predictions; whatever geometry makes those predictions possible is left for gradient descent to discover on its own, and it turns out to discover something that lines up with human intuitions about meaning.

The bridge to everything else on this site

Word2vec's real importance here is not the specific architecture — CBOW and skip-gram are, by later standards, simple — it is what it established as a habit of mind. Word2vec is the ancestor of every learned embedding in every model on this entire site: it is the moment a word, or later a token, stopped being an arbitrary symbol with an ID number and became a point in a continuous space that gradient descent could reason about. The query, key and value vectors of Lecture 1 begin their life as exactly this kind of learned point in space.

That is a large claim to rest on two shallow, non-attentional architectures from 2013, so it is worth being precise about what is and is not being claimed. Word2vec did not invent the idea of a distributed vector representation — earlier neural language models had used them too. What it did was make the idea cheap enough, at large enough scale, that a learned embedding stopped being a research curiosity and became the default starting point for representing a discrete symbol in almost every model built afterward, this site's included.