Word2Vec
The 2013 result that words could be numbers with meaningful geometry — the origin of embeddings, and its most famous demonstration was partly a trick.
When not to use it
- For anything current. One vector per word, forever. Contextual embeddings replaced it.
- On polysemous words. "Bank" gets one vector averaging river and money.
- Trained yourself. GloVe and fastText are pretrained and better.
- On unseen words. No representation at all. fastText fixes this with character n-grams.
Reach for something else instead
- Contextual embeddings — a word's vector depends on its sentence. The actual fix.
- Sentence/document embedding models — for retrieval, which is what you probably want.
- fastText — if you need static embeddings, this handles unseen words.
- GloVe — pretrained, count-based, comparable.
The full account
The demo that launched a thousand slides
king − man + woman = queen. It is the most reproduced result in the history of NLP, and for good reason: it appears to show that a model trained only on raw co-occurrence statistics has discovered gender as a direction in space, and that meaning obeys arithmetic. Subtract manness from king, get generic royalty, add womanness, get queen. It made embeddings feel like they were doing something profound rather than counting.
The same trick was then used for the opposite purpose. man is to computer programmer as woman is to homemaker. man is to doctor as woman is to nurse. The identical arithmetic, now demonstrating that these vector spaces had absorbed the prejudices of the text they were trained on — and that finding, from Bolukbasi and colleagues in 2016, launched an entire debiasing literature.
Both demos are doing something the audience is not told about.
The line of code nobody mentions
The analogy is evaluated with a function called 3CosAdd: compute king − man + woman, then find the nearest word vector to the result. Except that is not what the implementations do.
They exclude the input words from the candidate set. Nissim, van Noord and van der Goot spell out why in 2020: without that constraint, 3CosAdd would simply return one of the inputs whenever there isn't a closer neighbour. The vector king − man + woman lands closest to... king. So every implementation quietly removes king, man and woman from the list of possible answers, and then reports the nearest remaining word.
Queen is not what the arithmetic returns. Queen is what the arithmetic returns after you delete the actual answer.
The constraint is not a secret — Mikolov mentions it in the original paper. It is simply never mentioned again, in any of the thousands of slides that reproduce the result, and its effect is large: it is a strong constraint on both the semantic and the morphosyntactic analogies.
Which changes the bias finding too — and this is the delicate part
Nissim's title is the result. Man is to doctor as woman is to doctor. Run the analogy without the exclusion and the answer is doctor, because doctor is already the nearest vector. The stereotypical "nurse" appears only once you have forbidden the model from giving the correct, unbiased answer.
So some of the most-cited demonstrations of embedding bias are artefacts of an implementation detail. The authors demonstrate that several widely circulated biased analogies are not supported by the data.
It is critical not to over-read this, and the authors are explicit about it, so this page will be too: they are not saying embedding bias is fake. Their position is that human biases are genuinely present in word embeddings and genuinely need addressing — but that analogies are not an accurate tool for finding them, and the way they were used has exacerbated some possibly non-existent biases while hiding others. Their conclusion is the sentence this whole site is about: rather than striving to find sensational biases, observe the data as is, which is biased enough.
The bias is real. The famous evidence for it was theatre. Those are both true, and the second one is worse for the cause than admitting it, because a finding built on an artefact gets overturned and takes the real concern down with it.
| What people think it shows | What it shows | |
|---|---|---|
| king − man + woman = queen | arithmetic on meaning | queen is the nearest word after deleting king, man, woman |
| man:doctor :: woman:nurse | the embedding is sexist | without the exclusion, the answer is doctor |
| The offset does the work | a gender direction | Linzen: much of it is neighbourhood structure, not the offset |
| Debiasing fixes it | bias removed | Gonen & Goldberg: the geometry is hidden, not removed |
The offset may not even be the mechanism
Linzen made the deeper argument in 2016. If the vector offset is what produces the answer, then baselines that ignore the offset should fail. He tested them — and they do surprisingly well. Simply returning a nearest neighbour of one input word, with no arithmetic at all, solves a good fraction of the standard analogy set.
Which means much of what the analogy task measures is irrelevant neighbourhood structure: queen is simply close to king in the space, for the ordinary reason that royalty words co-occur with royalty words. The offset arithmetic is riding on top of a similarity structure that was already going to give you roughly the right answer, and getting the credit.
And the fix didn't work either
The Bolukbasi debiasing paper proposed identifying a gender subspace and projecting it out. Gonen and Goldberg tested that in 2019 and titled the result Lipstick on a Pig: the debiasing removes the visible symptom — the gendered direction that bias tests look along — while leaving the underlying geometry intact. Words that were clustered by gender before are still clustered by gender after; you can recover the information trivially. The metric improved and nothing else did.
So the arc is: a striking demo that depends on a hidden exclusion, a bias finding that depends on the same exclusion, a debiasing method that fixes the measurement rather than the model, and a critique of that debiasing that arrived three years later. Each step was published, cited, and built upon before anyone checked the one below it.
What survives
Word2vec was a real advance and none of this touches it. Dense vectors learned from co-occurrence genuinely capture semantic structure, genuinely beat sparse count vectors on downstream tasks, and genuinely started the line that leads to every embedding model in production today. Levy and Goldberg later showed it is implicitly factorising a shifted PMI matrix — which is a beautiful result and, in a way, the same lesson: the method was doing something older and more comprehensible than the magic it was sold as.
What does not survive is the slide. If you show king − man + woman = queen, say that the inputs were excluded from the answers. If you show a bias analogy, say the same. And if you want to measure bias in an embedding, use a tool built for it — Caliskan's WEAT — rather than the demo that was built to be impressive.
Further reading
- Mikolov et al. (2013), Efficient Estimation of Word Representations in Vector Space — the paper.
- Levy & Goldberg (2014), Neural Word Embedding as Implicit Matrix Factorization — it's PMI matrix factorisation in disguise.
- Bolukbasi et al. (2016), Man is to Computer Programmer as Woman is to Homemaker? — the bias findings, and read Gonen & Goldberg (2019) on why the debiasing didn't work.
- Nissim, van Noord & van der Goot (2020), Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor — Computational Linguistics; the analogy demos depend on an exclusion nobody mentions, in both directions.
- Linzen (2016), Issues in Evaluating Semantic Spaces Using Word Analogies — the neighbourhood structure, not the offset, is doing much of the work.
- Gonen & Goldberg (2019), Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings — the debiasing that was supposed to fix it doesn't.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Repeating the analogy demonstration uncritically. The evaluation excludes the input words; without that exclusion it returns
king. - Using static embeddings where context matters, which is most places.
- Thinking the neural framing was the innovation. Levy & Goldberg: it's implicit PMI matrix factorisation. The efficiency was the contribution.
- Assuming debiasing removed the bias. Gonen & Goldberg showed it mostly hid it.
At a glance
Often compared with
Where this sits
A destination. 7 concepts lead here, and nothing in the corpus depends on it.
Computed from the prerequisite graph, not assigned. How this works