Short answer. Because model capability tracks the volume and quality of text that existed in a language when the model was trained, and that distribution is extremely uneven. The effect is measurable on identical content: MMLU-ProX, which poses the same 11,829 questions in 29 languages, reports gaps of up to 24.3 points between high- and low-resource languages across 36 evaluated models, with strong models scoring above 70% in English and around 40% in Swahili. The dangerous part is not that output is obviously broken. Fluency degrades more slowly than accuracy, so the weaker-language output still reads well while being wrong more often — and English-only QA cannot see that, by construction.
This is the production consequence of the data gap rather than the classification behind it: how the gap shows up at each layer of a content pipeline, why it is invisible from headquarters, and how to tier a process around measured capability instead of applying one workflow everywhere.
Where the gap comes from
A model learns the distribution of the text it was trained on. Languages appear in that text in proportion to how much of the language was written down, digitised and reachable — not in proportion to how many people speak it. Those are very different quantities, which is why a language with hundreds of millions of speakers can have a fraction of the digital corpus of one with a tenth as many.
The consequence compounds through the stack:
- Tokenisation. Tokenisers trained predominantly on high-resource text split other languages into more tokens per unit of meaning, which costs context window and money.
- Instruction tuning. Instruction data is overwhelmingly English or translated from English, so the model's sense of what a good answer looks like is calibrated on English conventions.
- Evaluation. Suites were English-first, so regressions elsewhere were not visible during development.
- Retrieval. Fewer and lower-quality in-language sources exist to ground an answer on.
Each layer is individually reasonable. The accumulation is a large capability gap.
The distinction that matters operationally: fluency and accuracy degrade at different rates. Models learn the shape of a language from relatively little data, so output stays grammatical and idiomatic well past the point where factual reliability has dropped. A reviewer who does not speak the language sees fluent text and concludes it is fine. That is the single most expensive misreading in multilingual AI content.
What the measurements show
MMLU-ProX is the cleanest available evidence because it controls for content. It extends a reasoning-focused English benchmark into 29 typologically diverse languages using a semi-automatic translation process with expert validation, and each language version contains the same 11,829 questions — so a score difference between languages is a difference in the model, not in the difficulty of the questions. Across 36 state-of-the-art models, including reasoning-enhanced and multilingual-optimised ones, the authors report disparities of up to 24.3 points between high- and low-resource languages. The work was published at EMNLP 2025.
A second body of work addresses a subtler problem: for many languages the gap was not known, because no benchmark existed. A 2024 study created roughly one million human-translated words of new benchmark data across eight low-resource African languages — Amharic, Bambara, Igbo, Sepedi, Shona, Sesotho, Setswana and Tsonga — covering more than 160 million speakers, precisely because standard benchmarks did not exist and the gap was therefore unmeasured rather than small.
| Pipeline layer | Symptom in a low-resource language | What an English-only team sees |
|---|---|---|
| Generation | Higher factual error rate at unchanged fluency | Nothing. The output reads well |
| Tokenisation | More tokens per sentence; truncated context; higher cost | Unexplained cost variance between markets |
| Instruction following | Drift toward English rhetorical conventions | Text that is correct and reads as translated |
| Local knowledge | Wrong or missing market-specific facts, names, regulations | Confident text a local reader immediately distrusts |
| Evaluation | Regressions invisible to the test suite | Green dashboards and complaints from the regional office |
| Retrieval | Fewer, weaker in-language sources available | Thin answers attributed to a thin brief |
Reading down the third column explains why these programmes fail quietly. Every symptom is either invisible from headquarters or attributable to something other than model capability.
The commercial asymmetry
The gap would matter less if the affected languages were commercially marginal. The relationship runs the other way. CSA Research's 29-country survey of 8,709 consumers — reported via press release rather than as a published paper — found 76% of online shoppers prefer to buy with information in their own language and 40% will not buy from a site in another language at all. Preference for one's own language is not weaker in markets with lower English proficiency; it is stronger.
So the markets where models are least reliable are frequently the markets where publishing locally matters most for conversion. A programme that responds to weak model performance by defaulting those markets to English has chosen the worst-performing commercial option while appearing on the dashboard as a quality-conscious decision.
The alternative is not publishing unreviewed output either. It is tiering: accept a higher human cost per asset where the model is weakest, funded from the savings the same model produces where it is strong.
Building a pipeline that accounts for the gap
The goal is not a uniform process. It is a process whose intensity is set by measured capability per language, which requires measuring first.
- Tier the language list by measured capability, not by revenue. Run a small in-language evaluation per target language before committing to a workflow. Three tiers is enough: strong, adequate with review, and weak enough to require human authorship or heavy rewriting. Revenue determines whether you enter a market; capability determines how you produce for it.
- Build evaluation sets in-language, not translated. A translated test set measures translation, and it embeds English framing and English-relevant knowledge. Items authored by speakers, covering local conventions, entities and regulations, are the only way to see the gap that matters.
- Ground generation in retrieved in-language sources. Retrieval reduces reliance on what the model memorised, which is precisely what is thin here. Where in-language sources are scarce, retrieve from a trusted source in a strong language and translate under review — an explicit, reviewable step rather than a silent one.
- Constrain the output format more tightly where capability is weaker. Templates, controlled terminology and fixed structures reduce the space in which a model can be creatively wrong. Free-form generation is a luxury reserved for languages where the model is strong.
- Put the reviewer in-market, and give them a rubric. A fluent speaker abroad catches grammar; an in-market reviewer catches register, currency of usage and local factual error. Score against a defined error typology such as MQM so results are comparable across languages rather than a series of independent opinions.
- Sample inversely to capability. Applying the same 5% sample to English and to a weak-tier language accepts a much higher escape rate in the language you can least afford it in.
- Re-measure when models change. Capability in a given language can move substantially between versions, in either direction. Re-run the evaluation at each model change and re-tier — a language may have earned a lighter process, or lost one.
Three things teams believe that the evidence does not support
"Translate from English and the quality question is solved." It moves the question rather than solving it. Translation quality in low-resource pairs has the same underlying data problem, and pivoting through English introduces a documented class of errors around culture-specific content. Translation with review is a reasonable strategy; it is not a way to avoid needing review.
"The gap is closing, so this is temporary." Some of it is closing, unevenly. A language whose digital corpus is not growing benefits comparatively little from a larger training run, and reported gaps persist across frontier models. Planning on the gap disappearing is planning on someone else's roadmap.
"If it reads well, it is fine." This is the assumption the measurements most directly contradict. Fluency and accuracy decouple, and the decoupling is worst exactly where you can least verify it. It is also why a colleague's native-speaker spot check is not a substitute for a rubric-scored review by a qualified reviewer.
How Lifewood approaches this
Everything above is implementable in-house for a handful of languages. It stops being implementable somewhere between ten and twenty, and the reason is not technical: the process requires a qualified in-market reviewer available for every language on every release, indefinitely, plus a maintained in-language evaluation set per language. That is a staffing and network problem, and it is why multilingual programmes quietly contract to the languages a team can actually staff.
Lifewood operates that network as its core business: 50+ languages with region-native reviewers, 40+ delivery centres across 30+ countries, 56,788 registered contributors, and a dual-layer human-in-the-loop review process held to a 95%+ accuracy threshold.
The relevant point is narrower than a pitch: this problem is solved with people in the market, not with a better prompt, and any vendor answer that does not involve in-market reviewers is answering a different question. See multilingual data collection and AIGC services.
Sources and further reading
- MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation — 29 languages, 11,829 items each, 36 models; EMNLP 2025.
- "Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments", December 2024.
- CSA Research, "Can't Read, Won't Buy" third global survey — reported via press release rather than as a published paper.
- MQM Council, the MQM error typology.
- Companion guides: High-Resource vs Low-Resource Languages in AI Training and Beyond Translation: Why AI Needs Culturally Relevant Data.