Word2Vec
The 2013 result that words could be numbers with meaningful geometry — the origin of embeddings, and its most famous demonstration was partly a trick.
When not to use it
- For anything current. One vector per word, forever. Contextual embeddings replaced it.
- On polysemous words. "Bank" gets one vector averaging river and money.
- Trained yourself. GloVe and fastText are pretrained and better.
- On unseen words. No representation at all. fastText fixes this with character n-grams.
Reach for something else instead
- Contextual embeddings — a word's vector depends on its sentence. The actual fix.
- Sentence/document embedding models — for retrieval, which is what you probably want.
- fastText — if you need static embeddings, this handles unseen words.
- GloVe — pretrained, count-based, comparable.
Sources & further reading
- Mikolov et al. (2013), Efficient Estimation of Word Representations in Vector Space — the paper.
- Levy & Goldberg (2014), Neural Word Embedding as Implicit Matrix Factorization — it's PMI matrix factorisation in disguise.
- Bolukbasi et al. (2016), Man is to Computer Programmer as Woman is to Homemaker? — the bias findings, and read Gonen & Goldberg (2019) on why the debiasing didn't work.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Repeating the analogy demonstration uncritically. The evaluation excludes the input words; without that exclusion it returns
king. - Using static embeddings where context matters, which is most places.
- Thinking the neural framing was the innovation. Levy & Goldberg: it's implicit PMI matrix factorisation. The efficiency was the contribution.
- Assuming debiasing removed the bias. Gonen & Goldberg showed it mostly hid it.
At a glance
FieldDeep Learning
Year2013
The ideaa word is characterised by the company it keeps
What it enabledevery embedding you use
The famous resultreal, and it requires excluding the input words to work
What it actually isimplicit PMI matrix factorisation
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Word2Vec vs. contextual embeddings — one gives every word a single fixed vector for all time; the other gives it a vector that depends on the sentence it's in. That gap is a decade of progress.