RLHF (Reinforcement Learning from Human Feedback)
Training a model on human preferences rather than correct answers — the step that turned text predictors into assistants.
Reviewed July 10, 2026Stable
When not to use it
- When you have correct answers. If you can demonstrate the right output, supervised fine-tuning is simpler, cheaper, and more reliable. Preferences are for when correctness can't be written down.
- Without enough preference data. A reward model trained on a few hundred comparisons will confidently encode the noise.
- When labellers can't judge the task. If the model is better than the humans rating it, you're training toward their limits, not past them.
Reach for something else instead
- DPO — the same preference signal without the reward model or the RL machinery. The sensible default for most teams now.
- Supervised fine-tuning on good examples, when good examples exist.
- Prompting — a system prompt gets you a surprising amount of behaviour shaping for zero training.
Read more on the blog
- AI alignment and safety, without the hype or dismissalAI safety is discussed either as impending doom or as overblown hype, and neither framing helps you understand it. The actual landscape: what alignment means, the concrete risks experts agree on, the speculative ones they don't, and why serious people land in very different places.
- How AI models are trained: from raw text to a system that helpsA language model isn't programmed, it's grown, in stages, from a firehose of text into a system that answers helpfully. The full modern pipeline: pretraining, supervised fine-tuning, preference alignment, and the reasoning training that defines 2026, what each stage does, why none can be skipped, and how the recipe changed.
- What is reinforcement learning? Learning from rewardReinforcement learning went from a niche corner of AI obsessed with games and robots to the paradigm that shapes how every modern language model behaves. Here's what it actually is, learning by trial, reward, and consequence, why it's different from other machine learning, and how it quietly became the layer between a smart model and a useful one.
- Why AI hallucinates: the confident lie is a feature, not a bugAI models don't hallucinate because they're broken. They hallucinate because we trained and scored them in a way that rewards confident guessing over honest uncertainty, and that has a mathematical floor. The real mechanism, the 2026 research that pinned it down, and what actually reduces it.
Further reading
- Christiano et al. (2017), Deep Reinforcement Learning from Human Preferences — the technique, before language models.
- Ouyang et al. (2022), Training language models to follow instructions with human feedback — InstructGPT, the paper that made assistants work.
- Rafailov et al. (2023), Direct Preference Optimization — preference training without the reward model or the RL loop.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating the reward model as ground truth. It's an approximation, and hard optimisation against it finds its flaws rather than human preference.
- Ignoring the KL penalty's role, then wondering why the policy drifted somewhere strange.
- Assuming preference-trained means aligned. It means it produces outputs that labellers preferred — including sycophancy, hedging, and length.
At a glance
FieldLanguage & LLMs
Core ideatrain on preferences, not answers
PipelineSFT, reward model, policy optimisation
Weak linkthe reward model
Modern shortcutDPO
DifficultyAdvanced
Often compared with
RLHF vs. supervised fine-tuning — learning from which output was preferred vs. learning from the output that was correct.
Where this sits
27 concepts come first. Understanding it opens up 12 more.
13Levelsteps in
27Needs firstconcepts
12Opens up4% of 310
3Areascrosses fields
Learn these firstInstruction Tuning
Computed from the prerequisite graph, not assigned. How this works