Loss Function
The number that says how wrong the model is — and therefore the definition of what it's trying to become.
When not to use it
- As a stand-in for what you care about. You optimise cross-entropy; you probably want F1 or revenue or safety. Watch both, and don't confuse them.
- Unweighted on imbalanced data. It will learn to predict the majority class and report excellent numbers while being useless.
- As a comparison across runs with different setups. Loss values aren't comparable across different losses, batch sizes, or normalisations — the number alone is meaningless.
Reach for something else instead
- MAE or Huber instead of MSE when outliers shouldn't dominate. MSE squares errors and therefore worships them.
- Class weights or focal loss for imbalance, rather than resampling and hoping.
- Custom cost matrices when different mistakes genuinely cost different amounts — encode it rather than pretending they're equal.
Sources & further reading
- Lin et al. (2017), Focal Loss for Dense Object Detection — the standard answer to severe class imbalance.
- Clark & Amodei (2016), Faulty Reward Functions in the Wild — a boat spinning in circles for points. The clearest illustration of optimising the measure instead of the goal.
- Goodfellow, Bengio & Courville, Deep Learning, ch. 5–6 — the maximum-likelihood framing that explains why the standard losses are the standard ones.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Assuming the loss going down means the model is getting better at your task. It means it's getting better at the proxy.
- Adding terms until it works, without noticing you've created a hyperparameter for each one.
- Reporting a loss value as if it means something to a reader. It's a training signal, not a result.
At a glance
FieldDeep Learning
Core ideathe number the model minimises
Classificationcross-entropy
RegressionMSE, MAE, or Huber
Must bedifferentiable
DifficultyBeginner → Intermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Loss vs. metric — what the model optimises vs. what you actually judge it by. They are not the same and rarely move together.