Instruction Tuning

The full account

The step that turned a text predictor into something you can talk to

A base language model is a spectacularly good autocomplete and nothing more. Trained to predict the next token across the internet, it will happily continue your question with more questions, because on the web a question is often followed by other questions. Ask it "How do I reset my password?" and a base model might reply "How do I change my email? How do I delete my account?" — perfectly plausible text, useless as an answer. Instruction tuning is the step that fixes this, and it's a larger part of why modern models feel helpful than most people realise.

The intervention is conceptually simple: after pretraining, fine-tune the model on a dataset of instructions paired with good responses. Thousands to millions of examples of the shape here is a request, here is how a helpful assistant answers it. The model isn't learning new facts — those came from pretraining. It's learning a format, a disposition: that when text looks like a request, the continuation should be a direct, helpful completion of that request rather than a statistically likely ramble.

Why it works better than it should

The surprising empirical finding, sharpened by work like Zhou et al.'s "LIMA," is how little instruction data it takes. A base model instruction-tuned on only about a thousand carefully curated examples became strikingly capable at following instructions — which suggests the knowledge and ability were already latent in the pretrained weights, and instruction tuning mostly surfaces an ability the model already had rather than teaching it something new. The pretrained model already knew how to be helpful; it just didn't know that helpfulness was what you wanted. A small, high-quality set of demonstrations is enough to point it in the right direction.

This is the "superficial alignment" idea, and it reframes what instruction tuning is doing. It's not installing capability. It's selecting, from the vast space of behaviours a base model can produce, the narrow band that reads as a cooperative assistant — and the reason a thousand examples suffice is that you're choosing among existing behaviours, not building new ones.

Instruction tuning is not preference tuning

A common confusion worth clearing: instruction tuning and RLHF are different steps that do different things, usually applied in sequence. Instruction tuning uses supervised learning on demonstrations — here's a good answer, imitate it. It teaches the model to follow instructions at all. Preference-based methods that come after (RLHF, DPO) then optimise which of the model's acceptable answers humans prefer — tone, safety, helpfulness, honesty — by learning from comparisons rather than demonstrations.

The order matters and the division of labour matters. Instruction tuning gets you a model that answers the question. Preference tuning gets you a model that answers it the way people want. Skipping the first makes the second nearly impossible, because you can't rank responses from a model that doesn't produce coherent responses to rank.

The quality trap

Because so little data has such large effect, instruction tuning is unusually sensitive to what's in the examples. Quality dominates quantity: a thousand clean, diverse, genuinely-helpful demonstrations beat a hundred thousand mediocre ones, and bad instruction data teaches bad habits with the same efficiency that good data teaches good ones. Style leaks — if your demonstrations are all verbose, the model becomes verbose; if they hedge, it hedges. The model is learning a persona from the examples, and it learns whatever persona you actually showed it, not the one you intended.

This is why instruction datasets are curated, not scraped, and why the composition of that dataset is one of the quiet, high-leverage decisions in building a model that people find pleasant to use.