Multilingual AI
How language models behave outside English — where the capability comes from, why it degrades, and why the same sentence can cost four times as much in one language as another.
Reviewed July 15, 2026Stable
When not to use it
- As an assumption that English benchmark scores transfer. They do not, and the gap widens with resource scarcity.
- As a substitute for a dedicated translation system when translation is the actual task and quality is critical.
- As a claim that a model is culturally competent in a language because it is fluent in it.
Reach for something else instead
- Dedicated machine translation for translation specifically, where specialised systems still compete well.
- Regionally trained models, which now exist for several language families and often beat larger general models on their targets.
- Continued pretraining on target-language corpora, which is the durable fix where budget allows.
Read more on the blog
- Why AI works worse in your languageThe gap between English and everything else is not about linguistic difficulty. It is about training data share, tokenizer fitting, and where instruction-tuning stopped, and only one of those three is expensive to fix.
- AI in translation: the field that retired its own metricTranslation has fifty years of formal evaluation practice. In 2022 its own shared task published under the title "Stop using BLEU", retiring the metric the field had run on for twenty years.
Further reading
- Conneau et al. (2020), Unsupervised Cross-lingual Representation Learning at Scale — the XLM-R work establishing large-scale cross-lingual transfer.
- Joshi et al. (2020), The State and Fate of Linguistic Diversity and Inclusion in the NLP World — the taxonomy of language resource inequality.
- Ahia et al. (2023), Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models — measures the token-count penalty across scripts.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Evaluating in English and assuming the result holds elsewhere.
- Budgeting tokens using English as the unit, then finding costs three to four times higher in production.
- Treating fluency as evidence of accuracy. Non-English output is often fluent and factually wrong in ways that require a native speaker to catch.
- Assuming instruction-following transfers as well as comprehension does. It usually does not.
At a glance
FieldLanguage & LLMs
Drivescost, context length, quality outside English
Main constrainttokenizer and training-data share
Cost penaltyoften 3 to 4 times for non-Latin scripts
Fixregional models or continued pretraining
DifficultyPractical
Where this sits
25 concepts come first. Understanding it opens up 1 more.
11Levelsteps in
25Needs firstconcepts
1Opens up0% of 310
2Areascrosses fields
Computed from the prerequisite graph, not assigned. How this works