Why AI works worse in your language

A model that was trained overwhelmingly on English can answer in Japanese, translate into Swahili and write code comments in Portuguese without anyone teaching it those languages as separate tasks. That is surprising, and it is real. It is also, for most of the world's languages, not good enough, and the reasons it is not good enough are more mundane and more fixable than they first appear.

The performance gap between English and other languages has almost nothing to do with how hard those languages are. It comes from three things: how much of that language existed in the training data, how efficiently the tokenizer handles its script, and whether instruction-tuning covered it at all. Two of those are accidents of how the systems were built rather than facts about language, which means they are correctable, and one of them is quietly making non-English use three to four times more expensive.

This is an explanation of where multilingual capability comes from, why it degrades, what the degradation actually costs, and which of the fixes are real.

The gap, measured

It helps to be concrete about size, because "works less well" understates it considerably.

On enterprise retrieval tasks, accuracy drops of up to twenty-nine percent have been measured between English and non-English queries against the same underlying documents. That is not a subtle degradation. That is a system that answers correctly nine times in ten in English and roughly six or seven times in ten elsewhere, with the same corpus and the same model.

At the far end of the resource distribution it stops being a gap and becomes a floor. Evaluations on indigenous languages of the Americas, absent from the training data of the multilingual models tested, have produced accuracy around thirty-eight percent on a three-way classification task. Chance is thirty-three. The model is not performing poorly in those languages; it is performing at approximately the level of guessing.

Between those two poles sits most of the world. The pattern is consistent: performance tracks resource availability, and resource availability tracks how much of a language exists in scrapeable text on the public internet, which correlates with wealth and with colonial history rather than with speaker population. Bengali has more speakers than German and a fraction of the digital footprint.

Why speaker count does not predict anything

It is worth dwelling on the mismatch, because it drives most of the inequality.

Bengali has roughly a quarter of a billion speakers and a digital footprint smaller than that of several European languages with a tenth the population. Hausa, Yoruba and Igbo together cover a substantial share of West Africa and appear in scraped corpora at a rate that would suggest they are minority languages spoken by hobbyists. Meanwhile Icelandic, with a few hundred thousand speakers, is comparatively well served because Iceland is wealthy, connected, and has invested deliberately in language technology.

What the corpus measures is not how many people speak a language. It measures how much text in that language has been published on the open web, which is a function of internet penetration, literacy in the written standard, the existence of a publishing industry, whether the language has a settled orthography, and whether its speakers write in it online or code-switch into a colonial language when they type.

That last one is underappreciated. In many multilingual societies people speak one language and write another, or write their own language in a romanised form that no standard corpus recognises. The digital record systematically understates what people actually speak, and the models inherit that understatement.

What actually predicts performance

Three factors, none of which is linguistic complexity.

Training data share. The dominant one. A language's representation in the pretraining corpus is roughly its representation in scraped web text, which is wildly unequal. The standard taxonomy of this inequality sorts the world's languages into tiers, and the top tier contains a handful of languages while the bottom contains thousands with almost nothing.

Tokenizer efficiency. Covered in detail below, because it is the least understood and the most immediately expensive.

Instruction-tuning coverage. Pretraining is broad; the instruction-tuning that turns a text predictor into something that follows requests is usually much narrower. This produces a specific and confusing failure: a model that clearly understands your language, and answers a question in it fluently, while ignoring the actual instruction you gave. Comprehension transferred. Instruction-following did not.

What is not on this list is worth stating. Morphological complexity, script direction, agglutination, tonality: none of these reliably predict how well a model performs. Finnish and Turkish are morphologically demanding and models handle them reasonably, because there is enough Finnish and Turkish text. The difficulty is a data problem wearing a linguistics costume.

The tokenizer tax

This is the part that surprises people who have been working with these systems for years, and it is the clearest example of a fixable accident being mistaken for a property of language.

A model does not read characters. It reads tokens, produced by splitting text using a vocabulary of subword pieces learned from a training corpus. The algorithm counts which adjacent pairs of characters appear together most often and merges them into single units, repeating until the vocabulary reaches its target size. Common English words end up as single tokens. Everything else gets fragmented.

Because the merges were learned from text that was overwhelmingly English, the resulting vocabulary encodes English efficiently and everything else poorly. The consequence is a direct multiplier on cost and a direct reduction in usable context. The same sentence, carrying the same meaning, can consume three or four times more tokens in one language than another, and in the worst cases considerably more.

Three groups are hit hardest. Non-Latin scripts, because their characters were rare in the fitting corpus and often decompose into multiple bytes before any merging happens. Agglutinative languages, which build long words from many morphemes, where a single semantic unit shatters into many pieces. And romanised text, which is neither the original script nor standard English, and falls into a gap between the two distributions the tokenizer actually learned.

The costs compound in three directions at once. You pay more per request. Your effective context window shrinks by the same factor, so a model advertising a large context offers substantially less of it to some users than others. And performance itself degrades, because the model sees the text at a coarser effective resolution, with meaning spread across more positions than the equivalent English.

Systematic evaluation across large numbers of languages has found that tokenizer quality is a strong predictor of downstream task performance, independent of everything else. That is a striking result. A preprocessing decision, made once before training and frozen thereafter, accounts for a meaningful share of how well a model works in a given language.

The mitigation that has actually been adopted is larger vocabularies. Recent model families have moved to vocabularies several times bigger than the earlier norm, which leaves room for more non-English merges and reduces fragmentation. This helps and does not solve it, since vocabulary space is finite and the allocation still follows the training distribution.

Where the capability comes from

Given all that, the surprising thing is that cross-lingual performance is as good as it is. A model never explicitly taught to translate can translate. A model fine-tuned on a task in English often improves at that task in other languages it was never fine-tuned on.

The leading account is that these models develop a partially shared representation space. Semantically equivalent sentences from different languages end up in similar regions of the model's internal state, with language-specific processing concentrated near the input and output and more abstract, less language-bound processing in the middle. Evidence comes from probing studies that find parallel sentences clustering together, and from the transfer effect itself, which is hard to explain otherwise.

This has a practical implication people underuse: prompting in English and requesting output in the target language frequently beats prompting in the target language. The instruction is understood more reliably in the language the instruction-tuning covered, and generation into the target language is a separate and better-supported capability. It feels wrong and it often works.

There is a serious caveat, which the frontier section returns to. Recent work has found that stronger representational alignment between languages does not consistently produce better performance. If sharing a representation space were the whole mechanism, more sharing should mean more transfer, and it does not straightforwardly. Something is going on that the simple picture does not capture.

Where it breaks

The failures cluster in specific places rather than being uniform degradation.

Cultural and local knowledge. A model can be fluent in a language and ignorant of the world that language describes. Local institutions, regional history, culturally specific terms, the entities that matter in that context: these appear in proportion to their presence in the training data, which for most languages means barely. Translation-based evaluation misses this entirely, because a translated benchmark tests whether the model handles the language, not whether it knows the place.

Reasoning tasks specifically. The gap is larger on multi-step reasoning than on factual recall, which suggests the model's internal reasoning process is more anglophone than its knowledge is. Interestingly, this pattern does not hold uniformly: on some factual and real-world query distributions the gap narrows or behaves differently, which complicates any simple story.

Fluency without accuracy. This is the failure mode most likely to reach production undetected. Non-English output is often perfectly fluent and confidently wrong. Fluency is cheap for these models; correctness in a low-resource language is not. A reviewer who does not speak the language cannot tell the difference, and automated metrics that reward fluency will not catch it either.

Safety and moderation. Guardrails trained on English content transfer imperfectly. This runs in both directions, producing both under-blocking in languages the safety training did not cover and over-blocking of ordinary speech that pattern-matches to something flagged in English.

The evaluation problem underneath all of this

A quieter difficulty runs beneath every claim in this piece: we are not very good at measuring the thing.

Most multilingual benchmarks are translated from English. That makes them comparable across languages, which is why they are built that way, and it means they test a narrow thing: whether the model handles the language, holding the content constant. They cannot test whether the model knows anything about the world that language is spoken in, because the content came from somewhere else.

The result is a systematic blind spot. A model can score respectably on a translated benchmark in a language while being ignorant of that language's context, and the benchmark will not notice, because nothing in it asks about local institutions, regional history, or the entities that actually matter to speakers.

Benchmarks written natively in each language would fix this and are far more expensive to build, since they need domain expertise and cultural knowledge per language rather than one dataset translated many times. Several groups are building them. There are not many yet, and the languages that most need them are the ones least likely to get them, for the same resource reasons that caused the original problem.

This means the numbers quoted anywhere, including in this article, are probably optimistic. They measure the part of the gap that translated benchmarks can see.

Two strategies, and they disagree

There are broadly two responses to all of this, pursued by different groups with different assumptions.

Massively multilingual models aim to cover a hundred or more languages in one system, on the argument that scale plus transfer lifts everything, including languages with little data of their own. This is mostly the approach of the largest labs. Its advantage is that a language with almost no data still gets something, carried by transfer from related languages and from the shared representation.

Regional multilingual models train on ten to twenty related languages, usually built by academic groups, governments and non-profits closer to the languages in question. The argument is that shared capacity is zero-sum, that dominant languages crowd out the rest, and that a smaller model focused on a language family will beat a much larger general model on those languages. The evidence increasingly supports this for the languages concerned.

These are not merely technical positions. The massively multilingual approach concentrates capability in a small number of large organisations. The regional approach distributes it, and puts model development in the hands of people who speak the languages and know what cultural competence would mean. That is a real difference in who gets to decide what these systems know.

Worth noting alongside both: for translation specifically, dedicated translation systems still outperform general-purpose language models on low-resource pairs. The general model is more flexible and the specialised system is more accurate, which is the usual shape of that trade.

What to do about it

For anyone deploying in more than one language, the practical sequence is fairly short.

Evaluate in the target language, with native speakers. Not translated benchmarks, which test the wrong thing, and not automated metrics alone, which reward the fluency that masks the problem. This is the step most often skipped and it is the one that catches the failures that matter.

Budget tokens in the actual language. Estimating cost and context in English and then deploying in Hindi or Amharic produces a surprise of several times, in both directions at once, since the context window shrinks as the bill grows.

Try English instructions with target-language output. It costs nothing to test and it frequently wins.

Check instruction-following separately from comprehension. They fail independently, and a model that clearly understands the language may still ignore the format you asked for.

Consider a regional model. For several language families these now exist and outperform larger general models on their targets, often at much lower cost.

Where it matters enough, continue pretraining. Taking an open-weight model and continuing training on target-language corpora is the durable fix. It is expensive, and it is what actually closes the gap rather than working around it.

What is unresolved

Two questions lack answers, and one of them matters more than it initially appears.

Is multilingual competence transfer, or translation through an anglophone frame? The optimistic reading is that these models learn language-independent concepts and express them in whichever language is asked for. The less comfortable reading is that they reason in something English-shaped and translate at the edges. The finding that better representational alignment does not reliably improve performance is a data point against the simple version of the optimistic reading.

The distinction is not academic. If the second account is closer to true, then a model addressed in Yoruba is reasoning about the world through a frame built from anglophone text, and the resulting bias would be invisible to every evaluation currently in use, because those evaluations test whether the output is correct rather than whether the reasoning route was neutral. Fluent, accurate, and culturally displaced is a failure mode with no established name and no established test.

Whether tokenization should exist at all. The costs it imposes are unevenly distributed and are an artefact of a preprocessing choice rather than anything necessary. Byte-level and character-level models remove the artefacts at the cost of much longer sequences, which architectures with better-than-quadratic scaling make more affordable than they were. There is active work on learned and dynamic segmentation. Nothing has displaced subword tokenization at the frontier, and the case for doing so rests on whether the affected populations are considered enough of a priority to pay the efficiency cost.

The honest counter-argument

A piece that treated the multilingual gap as a straightforward injustice would be missing something, so it is worth stating the other side.

Training data follows text, and text follows who writes. A model trained on what exists cannot be blamed for the distribution of what exists, and the alternative, weighting scarce languages far more heavily, has its own costs in quality and can produce a model that is mediocre everywhere rather than good somewhere. There is also a reasonable position that translation-based access, using a strong English model with good translation at the edges, serves many users adequately at a fraction of the cost of language-specific development.

Against that: the tokenizer penalty is not a data problem, it is a design choice with a straightforward if expensive fix, and charging some users three times more for the same service is difficult to defend as an inevitability. And the cultural knowledge gap is not solved by translation, since translating a question about local institutions into English does not conjure knowledge the model never had.

Both positions are held by people who have thought about it carefully, and the disagreement is mostly about what is owed rather than about what is true.

The short version

The performance gap between English and other languages in AI systems is driven by three factors, none of which is linguistic difficulty: share of the training data, tokenizer efficiency for the script, and whether instruction-tuning covered the language. Measured gaps run to twenty-nine percent on enterprise retrieval tasks and, for languages absent from training entirely, to accuracy barely above chance.

The tokenizer penalty is the least understood and the most immediately costly. Because subword vocabularies are fitted on English-heavy corpora, other scripts fragment into three to four times more tokens, which multiplies price, shrinks the usable context window and degrades quality simultaneously. Non-Latin scripts, agglutinative languages and romanised text are hit hardest, and tokenizer quality independently predicts downstream performance. Larger vocabularies help and do not solve it.

Cross-lingual capability comes from a partially shared representation space, which is why prompting in English and requesting output in the target language often outperforms prompting in the target language. Failures cluster in cultural knowledge, multi-step reasoning, and fluent-but-wrong output that only a native speaker catches. Two strategies compete: massively multilingual models covering a hundred languages, and regional models covering ten to twenty, with the regional approach increasingly winning on its targets and distributing capability more widely.

The idea worth keeping is that most of this gap is an artefact of how the systems were built rather than a fact about languages, which means it is correctable, and the question of whether anyone corrects it is a question about priorities rather than about feasibility.

Common questions

Why do AI models perform worse in languages other than English? Three reasons, none of them linguistic difficulty. The share of that language in the pretraining corpus, which roughly tracks its presence in scraped web text and correlates with wealth rather than speaker numbers. The efficiency of the tokenizer for that script, since vocabularies fitted on English-heavy text fragment other scripts into far more pieces. And whether instruction-tuning covered the language, which is usually much narrower than pretraining and produces models that comprehend a language while ignoring instructions given in it.

Why does the same prompt cost more in some languages? Because models read subword tokens rather than characters, and the token vocabulary was learned from a training corpus dominated by English. Common English words are single tokens; other scripts fragment into many. The same sentence can therefore consume three to four times more tokens in one language than another, which multiplies the bill and reduces the usable context window by the same factor. It is a consequence of how the tokenizer was fitted, not a property of the language.

How can a model answer in a language it was barely trained on? Through cross-lingual transfer. These models appear to develop a partially shared representation space, where semantically equivalent sentences from different languages occupy similar regions of the model's internal state, with language-specific processing concentrated near the input and output. Learning something in one language therefore makes it partly available in others. The effect is real and it weakens as you move further from the languages that dominated the training data.

Which languages are affected worst by tokenization? Three groups. Non-Latin scripts, whose characters were rare in the fitting corpus and often decompose into multiple bytes before any merging. Agglutinative languages, which build long words from many morphemes that shatter into separate tokens. And romanised text, which is neither the original script nor standard English and falls into a gap between the two distributions the tokenizer actually learned.

Should I prompt in English or in the target language? Test both, and do not assume the target language wins. Prompting in English and requesting output in the target language frequently outperforms prompting in the target language, because instruction-following was trained mostly in English while generation into other languages is separately and better supported. It feels counterintuitive and it is worth measuring rather than assuming.

Are regional models better than large multilingual ones? For the languages they target, increasingly yes. Regional models train on ten to twenty related languages and often beat much larger general-purpose models on those languages at a fraction of the size. Massively multilingual models cover a hundred or more and offer something to languages with almost no data of their own via transfer. The disagreement is about whether shared capacity is zero-sum, and it is also a disagreement about who develops the models and therefore who decides what cultural competence means.

How should I evaluate a model in another language? With native speakers, on tasks in the actual target language, not on translated benchmarks. Translated benchmarks test whether the model handles the language rather than whether it knows the world that language describes, and they miss the most dangerous failure mode: output that is fluent, confident and factually wrong about local context. Automated metrics reward fluency and will not catch it either. Evaluate instruction-following separately from comprehension, since they fail independently.

Is the multilingual gap fixable? Partly, and the parts differ. The tokenizer penalty is a design choice with a known if expensive fix, including larger vocabularies and language-specific or byte-level approaches. Instruction-tuning coverage is a matter of collecting and curating data in more languages, which is work rather than research. The training-data share problem is harder, since it reflects what has been written and published. Continued pretraining on target-language corpora closes the gap durably and costs real money. None of it is blocked on unsolved science, which makes the remaining gap a question of priorities.