In-Context Learning
A model picking up a task from examples in the prompt, without any training — and the evidence that it isn't learning the task at all.
When not to use it
- To teach genuinely new knowledge. It locates existing ability; it doesn't add any. That's what fine-tuning is for.
- When a clear instruction would do. On instruction-tuned models, zero-shot often beats few-shot. Try it first.
- When you're curating examples for correctness. Format and label coverage matter more than the labels being right.
- When example order is doing the work. If permuting them swings your results, you have fragility, not a technique.
Reach for something else instead
- Zero-shot with a good instruction — often better on modern models, and free.
- Fine-tuning — when you need behaviour changed, not located.
- RAG — when the problem is missing facts, not missing format.
- Many-shot — if you have the context budget; it does keep helping.
Sources & further reading
- Brown et al. (2020), Language Models are Few-Shot Learners — where it arrives, and the name that may be wrong.
- Min et al. (2022), Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? — random labels barely hurt. The result that reframes prompting.
- Olsson et al. (2022), In-context Learning and Induction Heads — the most concrete mechanistic account.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Believing the examples teach the task. Random labels barely hurt.
- Expecting it to add knowledge. It selects from what pretraining put there.
- Ignoring format consistency while agonising over example choice. The format is what's transmitted.
- Not trying zero-shot first. Instruction tuning absorbed most of what few-shot was for.
At a glance
FieldLanguage & LLMs
What it istask performance from prompt examples, no weight updates
The key resultrandom labels barely hurt (Min et al., 2022)
What examples actually conveyformat, label space, input distribution
Best mechanistic accountinduction heads
Probably misnamedit's task location, not learning
DifficultyBeginner
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
In-context learning vs. fine-tuning — one selects an ability the model already has, temporarily; the other changes the weights. Only the second adds anything.