Reward Function
The number that tells an agent what you want — and the hardest thing to write correctly in all of AI.
When not to use it
- (You need one. The question is when yours is wrong.)*
- Before asking what the cheapest way to maximise it is. That's what you'll get, not what you meant.
- With hand-written dense shaping. If it isn't potential-based, you've changed the optimal policy.
- Sparse, on a hard exploration problem. Honest and unlearnable.
- As a proxy for something you actually care about. The agent optimises the proxy, exactly.
Reach for something else instead
- RLHF — learn it from comparisons. Moves the problem to a reward model that gets exploited.
- Verifiable rewards — only where the answer is checkable. The version that works.
- Inverse RL — infer from demonstrations. Not identifiable; your assumptions decide.
- Imitation learning — skip the reward, copy the expert. Can't exceed them.
Sources & further reading
- Ng, Harada & Russell (1999), Policy Invariance Under Reward Transformations — potential-based shaping; the one safe way to add hints.
- Ng & Russell (2000), Algorithms for Inverse Reinforcement Learning — infer reward from behaviour, and the reward isn't identifiable.
- Amodei et al. (2016), Concrete Problems in AI Safety — reward specification as the practical safety problem, before it was fashionable.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Adding shaping rewards that aren't potential-based, and silently redefining the task.
- Rewarding a proxy and expecting the intent. You get the proxy.
- Assuming a bug in the agent when the agent did exactly what you wrote.
- Thinking IRL solves specification. The reward isn't identifiable — the assumptions choose it.
At a glance
FieldFoundations
What it isyour entire specification of intent, as a scalar
The guaranteethe agent finds the cheapest route to maximise it
The theorem to knowpotential-based shaping leaves the optimal policy unchanged; nothing else does
The honest tensionhonest rewards are unlearnable, learnable ones are lies
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Sparse vs. dense reward — one says only what you want and can't be learned from; the other says a lot you didn't mean and can be. Neither is safe, and shaping is only provably safe if it's potential-based.