Cross-Entropy
The cost of being wrong about a distribution — the loss function training most classifiers and every language model, and information theory's most-used export to ML.
When not to use it
- For regression with continuous targets. Cross-entropy is for distributions over classes or tokens; squared error and its kin fit continuous outputs.
- When you need calibrated probabilities out of the box. Low cross-entropy doesn't guarantee calibration; that needs separate treatment.
- With extreme class imbalance, unmodified. The dominant class can swamp the loss; reweighting or focal loss is often needed.
Reach for something else instead
- Focal loss down-weights easy examples for imbalanced detection and classification.
- Hinge loss (as in SVMs) optimises a margin rather than a probability.
- Mean squared error is the counterpart for continuous targets.
Worked example
Cross-entropy is the loss that trains most classifiers. It measures how surprised your model is by the true answer — and it explodes when the model is confidently wrong. Suppose the true class is "cat," and two models predict:
Good model — P(cat) = 0.9:
loss = −log(0.9) = 0.105 ← small
Bad model — P(cat) = 0.1:
loss = −log(0.1) = 2.303 ← 22× worse
Notice it only looks at the probability given to the correct answer. Now the punishing part — a model that's confidently wrong, P(cat) = 0.01:
The loss grows without bound as confidence in the wrong answer rises. That asymmetry is deliberate: it makes the model pay dearly for being sure and wrong, which is exactly the behaviour you want to train out.
Further reading
- Shannon (1948), A Mathematical Theory of Communication — the information-theoretic foundation cross-entropy rests on.
- Goodfellow, Bengio & Courville, Deep Learning — derives cross-entropy loss and its equivalence to maximum likelihood.
- Szegedy et al. (2016), Rethinking the Inception Architecture — introduced label smoothing as a cross-entropy regulariser.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Treating cross-entropy and KL divergence as interchangeable. Cross-entropy includes the data's own entropy; KL is only the excess.
- Expecting low loss to mean good calibration. It doesn't; a model can be confidently, cheaply wrong in aggregate.
- Ignoring a few catastrophic errors because accuracy looks fine. Cross-entropy is dominated by confident mistakes accuracy hides.
At a glance
Where this sits
A destination. 2 concepts lead here, and nothing in the corpus depends on it.
Computed from the prerequisite graph, not assigned. How this works