F1 Score

The full account

Why is it a harmonic mean, and why is β one?

F1 is the harmonic mean of precision and recall. Two questions follow that almost nobody asks, and both have uncomfortable answers.

The harmonic mean is chosen because it punishes imbalance. The arithmetic mean of precision 1.0 and recall 0.0 is 0.5, which would let a classifier that flags exactly one case and gets it right look mediocre rather than useless. The harmonic mean of 1.0 and 0.0 is 0. That is a defensible design choice and it is the strongest part of the metric.

The subscript is the problem. F1 is the β=1 member of a family, and β sets how much more you care about recall than precision. β=1 asserts they matter equally. That is not a mathematical fact — it is a claim about your costs, and it is almost always false. In cancer screening a false negative is a death and a false positive is an anxious afternoon; those are not equal. In spam filtering a false positive is a lost job offer and a false negative is a nuisance; also not equal. Every F1 you have ever reported encodes a value judgement you did not make and probably would not endorse if it were stated out loud.

Van Rijsbergen, who introduced the measure in 1979, was explicit that β was a parameter for the user to set from their own preferences. The field fixed it at 1 and stopped mentioning it.

The cell that isn't there

The structural fact about F1 is easiest to see in the confusion matrix. There are four cells: true positives, false positives, false negatives, true negatives. Precision uses TP and FP. Recall uses TP and FN. F1 combines them.

No part of F1 touches the true negatives. All the cases your model correctly identified as negative — usually the overwhelming majority of the data — contribute nothing to the score. This is sometimes defended as a feature, because in information retrieval the true negatives are every document you didn't return, and counting them is meaningless. That defence holds in search. It does not hold when you're classifying patients, transactions or images, where getting the negatives right is the entire job most of the time.

It also means F1 is not symmetric: swap the labels of your two classes and the score changes. A metric whose value depends on which class you decided to call "positive" is measuring something about your naming convention.

Hand's argument, made twice, ignored twice

David Hand published a paper in 2009 arguing that AUC is not merely imprecise but incoherent: averaging across thresholds implicitly weights the cost of false positives against false negatives, those implied weights depend on the classifier's own score distribution, so comparing two models by AUC compares them under different cost assumptions for each model.

In 2018, with Peter Christen, he made the same argument about the F-measure. They show F can be rewritten as a weighted sum of precision and recall — and the weights depend on the classification method being evaluated. Their conclusion is stated plainly: the relative importance assigned to precision and recall should be an aspect of the problem and of the researcher, not of the particular method being used. It is the identical structural flaw, in the field's other default metric, nine years later, from the same author.

Both papers are well cited. Both metrics remain the default. The most interesting thing on this page is that the field has been told twice, by the same person, in clear language, and has not changed — because the alternative to a single convenient number is a conversation about costs, and nobody wants to have it before a leaderboard.

Macro or micro is a bigger decision than the metric

The figure above computes the part that catches people in production. With multiple classes you must average the per-class F1 scores, and there are two standard ways.

Micro pools every prediction across all classes and computes one F1 from the totals. Every example counts equally, so your large classes decide the number.

Macro computes F1 per class and averages the results. Every class counts equally, so a rare class with fifty examples has exactly as much say as a common one with fifty thousand.

Drag the rare class in the figure and watch macro fall to 0.64 while micro sits at 0.90 — same model, same predictions, same confusion matrix, two numbers 26 points apart. Neither is wrong. They answer different questions: micro asks how often you're right, macro asks whether you work for everyone. If the rare class is fraud, or a rare disease, or a minority dialect, macro is the number that matters and micro is the number that gets reported.

Papers frequently report "F1" without saying which. When you see a suspiciously high F1 on an imbalanced dataset, that is the first thing to check.

What to do

Report precision and recall separately. They are two numbers, both interpretable, and collapsing them into one throws away the only thing a reader needs to judge your model against their own costs.

If you must have one number, state β and justify it, or use MCC — it uses all four cells, it is symmetric under label swap, and it does not have the weighting flaw.

If you know your cost ratio, compute expected cost. This is what every one of these metrics is a proxy for, and it is the only one where the value judgement is yours and visible.

And always say macro or micro. On imbalanced data the gap between them is the finding, not a footnote.