Reward Hacking
An agent maximising your reward without doing what you wanted — not a rare bug, and there's a proof that you mostly can't design around it.
When not to use it
- (It's a failure mode. The question is when to expect it.)*
- Whenever your reward is a proxy. Which is always, except in verifiable domains. It's a theorem, not bad luck.
- Under heavy optimisation pressure. The proxy is fine when weakly optimised and breaks when pushed. That's the whole shape.
- With a capable agent and a loose environment. Better optimisers find more exploits.
- Watching the reward curve instead of the behaviour. CoastRunners had a beautiful reward curve.
Reach for something else instead
- (Mitigations, none of which solve it.)*
- Verifiable rewards — no proxy, no gap. Only works where answers are checkable.
- KL penalty / optimisation limits — a leash. Admits the proxy breaks under pressure.
- Process supervision — reward the reasoning; now you're specifying good reasoning.
- Constrained action space — it can't exploit what it can't reach.
Sources & further reading
- Clark & Amodei (2016), Faulty Reward Functions in the Wild — the CoastRunners boat; the canonical demonstration.
- Skalse et al. (2022), Defining and Characterizing Reward Hacking — unhackable proxies are essentially impossible. The result that reframes the problem.
- Gao, Schulman & Hilton (2023), Scaling Laws for Reward Model Overoptimization — the curve: true reward rises, peaks, then falls as you optimise the proxy harder.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating it as a bug in the agent. The agent maximised your reward exactly as instructed.
- Thinking a better-designed reward fixes it. Skalse et al.: unhackable proxies are essentially impossible.
- Watching reward curves rather than behaviour. The curve looks great while the boat is on fire.
- Missing that RLHF is this. Sycophancy and verbosity are what maximising a human-approval proxy looks like.
At a glance
FieldSafety & Ethics
The canonical examplethe CoastRunners boat, circling for power-ups, on fire, scoring 20% above humans
The theoremunhackable proxies are essentially impossible (Skalse et al.)
The shapeproxies work at low optimisation pressure and break at high
Its biggest deploymentRLHF; sycophancy is reward hacking
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Reward hacking vs. a bug — a bug is the system doing something you didn't specify; this is the system doing exactly what you specified.