Entropy
The average surprise of a distribution — a single number saying how uncertain an outcome is, and the quantity most of machine learning is built to reduce.
When not to use it
- As a measure over outcomes with no meaningful probabilities. Entropy needs a distribution; imposed on arbitrary data it's meaningless precision.
- As a proxy for model quality on its own. Low entropy means confident, not correct — a confidently wrong model has low entropy too.
- In continuous settings without care. Differential entropy can be negative and coordinate-dependent; usually KL divergence is the right tool there.
Reach for something else instead
- Variance captures spread for numeric variables, where entropy's "any outcome" generality is overkill.
- Gini impurity is a common, cheaper stand-in for entropy in decision trees with similar behaviour.
- KL divergence is the right measure when you care about the gap between two distributions rather than the uncertainty of one.
Worked example
Entropy measures average surprise, in bits. A fair coin (two equally likely outcomes) should come out to exactly 1 bit. Check it:
H = −(0.5·log₂0.5 + 0.5·log₂0.5)
H = −(0.5·(−1) + 0.5·(−1)) = 1 bit
Now a biased coin — 90% heads, 10% tails. It's more predictable, so entropy should drop below 1:
H = −(0.9·(−0.152) + 0.1·(−3.32))
H = −(−0.137 − 0.332) = 0.47 bits
Predictability halved the surprise. And a coin that always lands heads (p=1)? H = −(1·log₂1) = 0 bits — no surprise at all, because you already know the answer. That's the whole intuition: entropy is high when outcomes are uncertain, zero when they're certain.
Further reading
- Shannon (1948), A Mathematical Theory of Communication — defined entropy and proved the source coding theorem.
- Jaynes (1957), Information Theory and Statistical Mechanics — introduced the maximum-entropy principle.
- Cover & Thomas, Elements of Information Theory — standard reference for the properties and their use in learning.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Equating entropy with disorder. It's uncertainty about outcomes, a precise expectation, not a vibe.
- Assuming low entropy means a good model. It means a confident one, which is only good if it's also calibrated.
- Forgetting the base. Bits vs nats differ by a constant factor; mixing them silently corrupts comparisons.
At a glance
Where this sits
1 concept come first. Understanding it opens up 3 more.
Computed from the prerequisite graph, not assigned. How this works