Emergence
Abilities that appear suddenly at scale rather than improving gradually — the most cited claim about large models, and a NeurIPS best paper says it's a measurement artefact.
When not to use it
- As evidence of unpredictability, without checking the metric. That argument leaned on emergence and the evidence moved.
- When your metric is exact-match. You built the cliff. It isn't in the model.
- As a claim about the model's nature. The model improved smoothly; the scoring didn't.
- To dismiss all sharp transitions. Induction head formation is a real candidate, observed mechanistically.
Reach for something else instead
- Continuous metrics — edit distance, per-token accuracy, log-likelihood. The curve is smooth underneath.
- Mechanistic evidence — look inside. That's where real phase transitions have been found.
- Scaling laws — the smooth, predictable thing that was there all along.
Sources & further reading
- Wei et al. (2022), Emergent Abilities of Large Language Models — the claim, and the paper everyone cites.
- Schaeffer, Miranda & Koyejo (2023), Are Emergent Abilities of Large Language Models a Mirage? — NeurIPS Best Paper; it's the metric. Read both.
- Anderson (1972), More Is Different — what emergence means when it means something.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Citing emergence without citing Schaeffer. The rebuttal won Best Paper and is less known than the claim.
- Concluding the model changed abruptly. Your metric had a cliff; the model had a slope.
- Assuming no sharp transitions exist. Induction heads form abruptly — the evidence is internal, not behavioural.
- Forgetting exact match is often what users need. The experience is real; the explanation was wrong.
At a glance
FieldFoundations
The claimabilities appear abruptly at scale
The rebuttalit's the metric; discontinuous scoring creates the cliff (NeurIPS Best Paper, 2023)
The demonstrationswap to a continuous metric and emergence vanishes
What survivesinduction head formation, evidenced mechanistically
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Emergence vs. scaling laws — one says capability jumps unpredictably, the other says it improves smoothly. The metric decides which you see, and only one of them was ever in the model.