Reward Hacking
An agent maximising your reward without doing what you wanted — not a rare bug, and there's a proof that you mostly can't design around it.
Reviewed July 13, 2026Stable
When not to use it
- (It's a failure mode. The question is when to expect it.)*
- Whenever your reward is a proxy. Which is always, except in verifiable domains. It's a theorem, not bad luck.
- Under heavy optimisation pressure. The proxy is fine when weakly optimised and breaks when pushed. That's the whole shape.
- With a capable agent and a loose environment. Better optimisers find more exploits.
- Watching the reward curve instead of the behaviour. CoastRunners had a beautiful reward curve.
Reach for something else instead
- (Mitigations, none of which solve it.)*
- Verifiable rewards — no proxy, no gap. Only works where answers are checkable.
- KL penalty / optimisation limits — a leash. Admits the proxy breaks under pressure.
- Process supervision — reward the reasoning; now you're specifying good reasoning.
- Constrained action space — it can't exploit what it can't reach.
Read more on the blog
- The secret language that never was, and the escape that didThe famous stories about AI going rogue are mostly false. The verified incidents are less dramatic and more concerning, and the difference between them is the whole subject.
- What is AI sycophancy? Why AI tells you what you wantTell an AI its plan is brilliant and it agrees; push back on a correct answer and it caves. This is sycophancy, the tendency to tell you what you want to hear rather than what is true. It feels like a personality quirk, but it is the predictable result of how these models are trained, which is why it is so hard to remove.
- AI alignment and safety, without the hype or dismissalAI safety is discussed either as impending doom or as overblown hype, and neither framing helps you understand it. The actual landscape: what alignment means, the concrete risks experts agree on, the speculative ones they don't, and why serious people land in very different places.
- What is reinforcement learning? Learning from rewardReinforcement learning went from a niche corner of AI obsessed with games and robots to the paradigm that shapes how every modern language model behaves. Here's what it actually is, learning by trial, reward, and consequence, why it's different from other machine learning, and how it quietly became the layer between a smart model and a useful one.
Further reading
- Clark & Amodei (2016), Faulty Reward Functions in the Wild — the CoastRunners boat; the canonical demonstration.
- Skalse et al. (2022), Defining and Characterizing Reward Hacking — unhackable proxies are essentially impossible. The result that reframes the problem.
- Gao, Schulman & Hilton (2023), Scaling Laws for Reward Model Overoptimization — the curve: true reward rises, peaks, then falls as you optimise the proxy harder.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating it as a bug in the agent. The agent maximised your reward exactly as instructed.
- Thinking a better-designed reward fixes it. Skalse et al.: unhackable proxies are essentially impossible.
- Watching reward curves rather than behaviour. The curve looks great while the boat is on fire.
- Missing that RLHF is this. Sycophancy and verbosity are what maximising a human-approval proxy looks like.
At a glance
FieldSafety & Ethics
The canonical examplethe CoastRunners boat, circling for power-ups, on fire, scoring 20% above humans
The theoremunhackable proxies are essentially impossible (Skalse et al.)
The shapeproxies work at low optimisation pressure and break at high
Its biggest deploymentRLHF; sycophancy is reward hacking
DifficultyIntermediate
Often compared with
Reward hacking vs. a bug — a bug is the system doing something you didn't specify; this is the system doing exactly what you specified.
Where this sits
A destination. 33 concepts lead here, and nothing in the corpus depends on it.
15Levelsteps in
33Needs firstconcepts
0Opens upnothing further
1Areastays here
Computed from the prerequisite graph, not assigned. How this works