Entropy
The average surprise of a distribution — a single number saying how uncertain an outcome is, and the quantity most of machine learning is built to reduce.
Reviewed July 16, 2026Stable
When not to use it
- As a measure over outcomes with no meaningful probabilities. Entropy needs a distribution; imposed on arbitrary data it's meaningless precision.
- As a proxy for model quality on its own. Low entropy means confident, not correct — a confidently wrong model has low entropy too.
- In continuous settings without care. Differential entropy can be negative and coordinate-dependent; usually KL divergence is the right tool there.
Reach for something else instead
- Variance captures spread for numeric variables, where entropy's "any outcome" generality is overkill.
- Gini impurity is a common, cheaper stand-in for entropy in decision trees with similar behaviour.
- KL divergence is the right measure when you care about the gap between two distributions rather than the uncertainty of one.
Sources & further reading
- Shannon (1948), A Mathematical Theory of Communication — defined entropy and proved the source coding theorem.
- Jaynes (1957), Information Theory and Statistical Mechanics — introduced the maximum-entropy principle.
- Cover & Thomas, Elements of Information Theory — standard reference for the properties and their use in learning.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Equating entropy with disorder. It's uncertainty about outcomes, a precise expectation, not a vibe.
- Assuming low entropy means a good model. It means a confident one, which is only good if it's also calibrated.
- Forgetting the base. Bits vs nats differ by a constant factor; mixing them silently corrupts comparisons.
At a glance
FieldFoundations
FormulaH(X) = −Σ p log p
Maximised bythe uniform distribution
Minimised bya point mass (certainty)
DifficultyIntermediate