Q-Learning
Learning the value of every action in every state, by bootstrapping off your own estimates — which converges beautifully in theory and diverges in practice.
Reviewed July 13, 2026Stable
When not to use it
- With continuous actions. You'd have to maximise over a continuum every step. Use policy gradients.
- When samples are expensive. It needs tens of millions. That's why RL lives in simulation.
- Without a target network and replay. The deadly triad will diverge, not just underperform.
- Expecting the tabular guarantees. A neural network voids them entirely.
Reach for something else instead
- Policy gradient / PPO — continuous actions, and what LLM work uses.
- Model-based RL — far better sample efficiency, more machinery.
- Contextual bandits — if actions don't affect future states, this is much simpler.
- Imitation learning — if you have demonstrations, copying is cheaper than exploring.
Q-learning builds a table of how good each action is in each state.
The agent learns a Q-value for every state-action pair — the expected long-term reward of taking that action there — and picks the highest. It updates each estimate toward the reward it actually got plus the best it expects next (the Bellman update), gradually converging on the optimal policy through trial and error.
Further reading
- Watkins & Dayan (1992), Q-learning — the convergence proof; bootstrapping off your own estimates works.
- Mnih et al. (2015), Human-level control through deep reinforcement learning — DQN; Atari from pixels, and the two tricks that made it stable.
- van Hasselt, Guez & Silver (2016), Deep Reinforcement Learning with Double Q-learning — the max operator is biased upward; decouple selection from evaluation.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Expecting convergence with a neural network. The proof is for tables; the triad is right there.
- Skipping the target network. You're regressing toward a target that moves when you update.
- Ignoring overestimation. The max over noisy estimates is biased upward, systematically.
- Reaching for RL when a bandit fits. If your actions don't change the next state, this is enormous overkill.
At a glance
FieldFoundations
What it learnsthe long-run value of every action in every state
The trickbootstrap: update a guess using a guess
Why it's off-policythe max means you learn the greedy policy while behaving however
The deadly triadapproximation + bootstrapping + off-policy can diverge
DifficultyIntermediate
Often compared with
Tabular vs. deep Q-learning — one has a convergence proof, the other has the deadly triad and a bag of tricks. Only the second scales.
Where this sits
3 concepts come first. Understanding it opens up 1 more.
3Levelsteps in
3Needs firstconcepts
1Opens up0% of 310
1Areastays here
Learn these firstReinforcement Learning
Computed from the prerequisite graph, not assigned. How this works