AI in translation: the field that retired its own metric

TL;DR. Machine translation is the only domain in this series with a long formal evaluation tradition, and it is worth studying because it already ran the experiment everyone else is running now. In 2022 the field's own shared task published its results under the title "Stop using BLEU", retiring the metric the discipline had relied on for twenty years. Human parity was declared repeatedly and overturned each time, not by worse systems but by better evaluation: measured sentence by sentence the claim held, and measured at document level it did not. The field has now reached what researchers call a measurability ceiling, where metrics agree with human annotators about as much as annotators agree with each other. Every other AI domain is currently where translation was around 2005, using a surface-overlap proxy and calling it quality.

---

In 2002 a paper introduced a way to score a machine translation automatically by counting how many word sequences it shared with a human reference. It was fast, cheap, language-independent and good enough to rank systems, and it became the field's standard for two decades.

In 2022, the official results of the field's own metrics shared task were published under the title "Stop using BLEU."

That is not a critic's complaint or a blog post. It is the discipline's central evaluation forum announcing, in the title of its own findings paper, that the number everyone had been reporting for twenty years should no longer be used.

No other area of AI has done this. Every other domain in this series is still reporting its equivalent of BLEU, and translation is worth studying precisely because it is thirty years ahead and already knows how this goes.

What fifty years of measurement produced

The lineage is worth laying out, because each step was a genuine improvement and each one was eventually found insufficient.

Surface overlap, from 2002. Count shared word sequences against a reference. Cheap, reproducible, and it correlates with quality only while systems are bad. Once output becomes fluent, a translation can be excellent and share few exact sequences with the particular reference someone happened to write.

Linguistic features, from 2005. Add stemming and synonym matching so a different word choice is not punished. Better, and it depends on handcrafted resources that do not exist for most languages, which limits it to the languages that needed help least.

Character-level matching, from 2015. Score on character sequences instead of words. Language-agnostic and more robust to morphology, and still fundamentally surface comparison.

Embedding comparison, from 2020. Compare contextual representations rather than strings, so a paraphrase can score well when the meaning matches. A real conceptual advance, and it moves the problem into a model whose own judgement now needs validating.

Learned metrics, from 2020 onward. Train a model to predict human quality ratings directly. The current state of the art, and the best of them do fine-grained error detection rather than emitting a single number.

And a parallel human track. A structured error-annotation framework, published in 2014, has expert annotators mark errors by category and severity rather than giving a holistic score. It is expensive and it is the closest thing the field has to ground truth.

The pattern is the important part. Every automatic metric was adopted because it correlated with human judgement on the systems of its day, and every one degraded as systems improved past the range where it was validated.

The human parity claims, and why they kept failing

This is the part every other domain should read carefully.

Human parity in translation was announced more than once, on the strength of evaluations showing no significant difference between machine and professional human output. The claims were not fraudulent and the evaluations were real.

They were overturned by better evaluation rather than by worse systems.

The decisive finding was that parity held when raters judged isolated sentences and disappeared when they judged whole documents. A system can produce sentences that are individually indistinguishable from professional work while producing a document that is not, because the failures are in things a sentence cannot show: a term translated three ways across a page, a pronoun that loses its referent, a register that drifts, a discourse connective that inverts an argument.

The unit of evaluation was determining the answer.

Two further corrections followed. One examined whether the human comparison itself was strong enough, since a professional translator working quickly on unfamiliar material is not the ceiling the phrase "human parity" implies. Another produced a formal set of recommendations for how such claims should be assessed at all.

This is the single most transferable finding in this article, and it generalises directly. An agent whose steps each succeed 95% of the time is not a 95% agent. A support system with high per-response quality can produce a bad conversation. A summariser that gets every sentence right can misrepresent a document. Whenever a system is evaluated on units smaller than the thing it is actually used for, the evaluation flatters it, and the gap grows with the length of the real task.

The measurability ceiling

The most recent and least comfortable finding.

Learned metrics have become good enough that they now agree with expert human annotators at roughly the rate expert annotators agree with each other.

That sounds like success and it creates a specific problem. If a metric matches humans as closely as humans match each other, there is no headroom left in which to demonstrate that a better metric is better. Any further gain in measured agreement could be genuine quality discrimination, or it could be a metric learning the idiosyncrasies of a particular annotation protocol: which errors that pool of annotators happens to weight, how they use the severity scale, what they do with ambiguous cases.

Nothing in the measurement can distinguish those two. Researchers describe this as proximity to a measurability ceiling, and the response being proposed is harder benchmarks and more rigorous annotation rather than better metrics, because the metric side of the problem has run out of room.

The parallel with human judgement is exact and worth stating. Inter-annotator agreement is the ceiling on any metric validated against annotators. A field that reaches it has not solved measurement; it has exhausted the measurement approach it was using.

What translation actually got right

It is easy to read the above as failure. It is closer to the opposite, and four things distinguish this field from every other one in this series.

It published the retirement. A shared task that announces its own standard metric should be abandoned is doing something no other subfield has managed. The equivalent would be a benchmark organisation publishing that its leaderboard is misleading.

It maintained an expensive human standard alongside the cheap one. Structured expert error annotation is slow and costly and the field kept funding it, which is why the automatic metrics could be caught failing at all. A field with only automatic metrics cannot discover that its automatic metrics are wrong.

It ran an annual shared task with a fixed protocol for decades. Comparable results across years is what let anyone see the metrics degrading as systems improved. Most AI benchmarks are replaced before that pattern could become visible.

And it took its own parity claims apart. The papers reassessing human parity came from inside the field, promptly, with better methodology, and they were absorbed rather than resisted.

None of that is about translation. All of it is about how a field builds evaluation, and it is the reason this is the domain worth copying.

Where it is deployed, and where the failures land

Worth being concrete, because the evaluation story can obscure a technology that works.

High volume, low stakes. User-generated content, product listings, support articles, subtitles. Enormous scale, real value, and errors that cost a moment of confusion. This is most of the deployed volume.

Professional translation with a human editor. The dominant professional pattern is machine output post-edited by a qualified translator. Productivity gains are substantial and well documented, and the human is doing a different job from the one they did before rather than a smaller version of it.

Where it becomes dangerous. Clinical, legal and safety-critical settings, where a mistranslation can cause harm and the person relying on it cannot evaluate it. Research on clinical use has examined how physicians detect harm in machine-translated material, finding that quality estimation helps a reader calibrate how much to rely on output and that back-translation surfaces critical errors. Both are ways of giving the reader a signal about a translation they cannot themselves check, which is the actual problem in every high-stakes deployment.

And the language asymmetry runs through all of it. Quality tracks training data availability, so the languages with the least digital text get the worst output and the fewest evaluation resources. The metrics validated for high-resource pairs have not been validated for the pairs where errors matter most.

What this means for evaluating anything else

Six lessons, and they cost this field decades to learn.

Validate your metric against the systems you have now, not the ones it was designed for. Every metric here degraded as systems improved past its validation range. A benchmark that separated models well in 2023 may not separate them in 2026, and nothing announces when that happens.

Evaluate on the unit you deploy. Sentence-level parity vanished at document level. Whatever you measure in pieces will look better than the whole.

Keep an expensive human standard. It is the only thing that can catch a cheap metric going wrong, and the temptation to drop it grows exactly as the cheap metric becomes convincing.

Watch for the agreement ceiling. When your metric matches your annotators as well as they match each other, further improvement is unfalsifiable. That is a signal to change the measurement, not to celebrate.

Report the human comparison honestly. Parity against a rushed non-specialist is a different claim from parity against an expert working normally, and the phrase does not distinguish them.

And publish the retirement when it comes. The most valuable single act in this field's evaluation history was announcing that its own standard number should not be used.

What is unresolved

How to evaluate above the sentence. Document-level evaluation is understood to be necessary and is far harder to do consistently, because coherence, terminology and register are judgements that annotators disagree on more than they disagree about a single sentence.

Whether learned metrics can be validated at all past the ceiling. If inter-annotator agreement bounds what any metric can demonstrate, the field needs either better annotation or a different validation strategy, and neither exists yet.

What happens for low-resource languages. The metrics, the annotation frameworks and the reference corpora are all concentrated in the languages that already had them. Whether findings transfer is largely untested.

And whether general models change the picture. Large language models translate competently without being translation systems, which raises a question the field's protocols were not designed for: how to evaluate a system that was never optimised for the task and has no in-domain training signal to inspect.

The counter-argument

BLEU was retired because it succeeded, not because it failed. A metric that ranked systems reliably for twenty years, drove a generation of progress and was cheap enough to run on every experiment is one of the most successful measurement tools in computing. Discarding it once systems outgrew it is the tool working as intended, and framing that as an indictment mistakes an ending for a mistake.

Translation is an unusually favourable case for evaluation. There is a correct answer, or at least a bounded set of acceptable ones, and a reference can be written in advance. Most AI tasks have neither. Holding up a field whose problem is comparatively well-posed as the model for fields whose problems are not may not transfer as cleanly as it appears.

The parity corrections may over-correct. Document-level failures are real and the systems that produced sentence-level parity were still extraordinarily good. A reader who takes "parity was overturned" as "machine translation is not close to professional quality" has drawn the wrong conclusion from a methodological refinement.

And the measurability ceiling might be a temporary artifact. It reflects the current annotation protocol. Better protocols with clearer severity scales and more expert annotators could raise it, and describing the limit as fundamental when it may be procedural is premature.

The short version

Machine translation has fifty years of formal evaluation practice, and it already ran the experiment every other AI domain is running now.

In 2022 the field's own metrics shared task published its results under the title "Stop using BLEU", retiring after twenty years the surface-overlap metric the discipline had run on. No other area of AI has publicly retired its standard measure.

The lineage shows the same pattern five times. Surface overlap, linguistic features, character matching, embedding comparison, learned metrics: each was adopted because it correlated with human judgement on the systems of its day, and each degraded as systems improved past the range where it had been validated.

Human parity was declared and overturned repeatedly, by better evaluation rather than by worse systems. The decisive finding was that parity held when raters judged isolated sentences and vanished when they judged whole documents, because the failures live in things a sentence cannot show: inconsistent terminology, lost referents, drifting register. The unit of evaluation was determining the answer, and that generalises directly to agents evaluated per step, summarisers evaluated per sentence, and support systems evaluated per response.

The field has now reached a measurability ceiling, where learned metrics agree with expert annotators about as often as annotators agree with each other. Past that point, a better score could be genuine discrimination or could be overfitting to one annotation protocol, and nothing in the measurement separates them.

What makes this the domain worth copying is not its results but its practice. It published the retirement of its own metric. It kept funding expensive human annotation alongside the cheap automatic kind, which is the only reason the cheap kind could be caught failing. It ran a fixed annual protocol for decades so degradation became visible. And it dismantled its own parity claims promptly, from the inside.

Every other AI domain is currently around where translation was in 2005: reporting a surface proxy, treating correlation with human judgement as settled, and not yet aware that the correlation expires.

Common questions

Why did researchers stop using BLEU? Because it measures surface overlap with a reference translation, and that correlates with quality only while systems are poor. Once output became fluent, a translation could be excellent while sharing few exact word sequences with the particular reference someone wrote. The field's own metrics shared task published its 2022 results under the title "Stop using BLEU", stating that neural metrics were better and more robust, which is the clearest public retirement of a standard metric anywhere in AI.

Has machine translation achieved human parity? It was declared more than once and each claim was overturned by better evaluation rather than by worse systems. The decisive result was that parity held when raters judged isolated sentences and disappeared when they judged whole documents, because failures appear in consistency of terminology, pronoun reference and register, none of which a single sentence can reveal. Later work also questioned whether the human comparison was strong enough, since a professional working quickly on unfamiliar material is not the ceiling the phrase implies.

What replaced BLEU? Learned neural metrics trained to predict human quality ratings, the best of which perform fine-grained error detection rather than emitting one number. Alongside them the field maintained a structured human framework in which expert annotators mark errors by category and severity. That expensive human track is what made it possible to discover the automatic metrics were failing, since a field with only automatic metrics cannot find out its automatic metrics are wrong.

What is a measurability ceiling? The point where a metric agrees with expert human annotators about as often as annotators agree with each other. Past it, any further gain in measured agreement could be genuine quality discrimination or could be the metric learning the idiosyncrasies of one annotation protocol, and nothing in the measurement distinguishes them. Machine translation evaluation is described as being near this point, which is why the proposed response is harder benchmarks and better annotation rather than better metrics.

Why does document-level evaluation matter? Because it is the unit people actually use. A system can produce sentences that are individually indistinguishable from professional work while producing a document that is not, since terminology can vary across a page, pronouns can lose their referents and register can drift. This generalises well beyond translation: whatever is measured in pieces will look better than the whole, and the gap grows with the length of the real task.

Where is machine translation actually deployed? Mostly in high-volume, low-stakes settings such as user-generated content, product listings, support articles and subtitles, where scale is enormous and an error costs a moment of confusion. In professional work the dominant pattern is machine output post-edited by a qualified translator. The risky deployments are clinical, legal and safety-critical, where a mistranslation can cause harm and the person relying on it cannot evaluate it.

How do you use machine translation safely in high-stakes settings? By giving the reader a signal about output they cannot check themselves. Research on clinical use found that quality estimation helps a reader calibrate how much to rely on a translation and that back-translation surfaces critical errors. Neither makes the translation correct; both make its reliability visible, which is the actual requirement whenever the person depending on the output cannot read the source.

What can other AI fields learn from translation evaluation? Six things it took decades to establish. Revalidate metrics against current systems, since every metric here degraded as systems improved past its validation range. Evaluate on the unit you deploy. Keep an expensive human standard, because it is the only thing that catches a cheap metric going wrong. Watch for the agreement ceiling, past which improvement is unfalsifiable. State what the human comparison actually was. And publish the retirement when a metric stops working.