KL Divergence
A measure of how far one distribution is from another — not a distance, but the workhorse behind variational inference, RLHF, distillation, and diffusion.
Reviewed July 15, 2026Stable
When not to use it
- As a distance. It's asymmetric and violates the triangle inequality; treating it as a metric produces wrong reasoning.
- When supports don't overlap. If q is zero where p isn't, KL is infinite — Wasserstein or a smoothed variant is safer.
- For symmetric "how different are these two" questions. Use a symmetric measure (Jensen–Shannon, Wasserstein) when neither distribution is privileged.
Reach for something else instead
- Jensen–Shannon divergence symmetrises KL and stays finite.
- Wasserstein distance gives a true metric with a meaningful geometry, better-behaved for generative models.
- Total variation distance bounds how differently two distributions can weight any event.
Further reading
- Kullback & Leibler (1951), On Information and Sufficiency — the original definition.
- Blei, Kucukelbir & McAuliffe (2017), Variational Inference: A Review for Statisticians — how reverse-KL minimisation underlies modern approximate inference.
- Hinton, Vinyals & Dean (2015), Distilling the Knowledge in a Neural Network — distillation as matching distributions.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Calling it a distance. The asymmetry is the whole point and ignoring it leads to real errors.
- Forgetting the infinity. Zero probability under q where p is positive makes KL blow up — a common training instability.
- Mixing up the directions. Forward KL is mean-seeking (blurry), reverse KL is mode-seeking (narrow); they produce different models.
At a glance
FieldFoundations
FormulaΣ p log(p/q)
Symmetric?No
Zero whenp = q
PowersVAEs, RLHF, distillation, diffusion
DifficultyAdvanced
Where this sits
A destination. 2 concepts lead here, and nothing in the corpus depends on it.
2Levelsteps in
2Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstEntropy
Computed from the prerequisite graph, not assigned. How this works