How Multilingual AI Models Are Benchmarked

A useful framing from recent survey work identifies three core challenges in multilingual evaluation: coverage, meaning which languages are tested at all; representativeness, meaning whether the test reflects the language and culture rather than an English original; and trust, meaning whether the result is scientifically reliable. Translated benchmarks and US-centric framing were named as the two main representativeness deficiencies.

The practical consequence is that "supports 100 languages," "was evaluated in 100 languages," and "performs well in 100 languages" are three different claims, and the market routinely presents the first as though it were the third. A Microsoft-affiliated survey of multilingual evaluation found that in every world region a substantial share of languages are evaluated primarily through translated content, with Europe and East Asia the most translation-dependent. Sub-Saharan Africa showed the highest share of natively authored content, a direct result of language-specific benchmarks built from scratch by regional research communities. Buyers comparing multilingual data collection providers should treat "language coverage" as a marketing claim until it is paired with an evaluation methodology.

Translationese is the interference a source language leaves in a translated text — detectable tokens and syntactic structures that let a reader identify the original. The phenomenon is well documented in linguistics, and in an evaluation context it is a contaminant rather than a curiosity.

Cue inheritance is the mechanism by which a model scores well by recognising the shape of the original English question rather than understanding the target language. Because all non-English items derive from a common English source, model performance may reflect identification of preserved English cues rather than genuine comprehension. Earlier work demonstrated exactly this, showing that translation introduces subtle artifacts models exploit as non-semantic shortcuts.

Disagreement with speakers is the most decision-relevant finding: translated benchmarks align far worse with local human judgments than natively constructed alternatives, reported as a Spearman correlation of 0.47 against 0.68. If a benchmark and its users disagree, the benchmark is the thing that is wrong.

Translation quality also varies more than most buyers assume. Global-MMLU, a substantial effort covering 42 languages, combined machine translation with crowdsourced human verification, but only around 20% of machine-translated texts underwent manual correction, and questions and answers were translated separately, producing observable grammatical inconsistencies in some languages. This is one reason multilingual LLM training data quality depends on native review, not just translation coverage.

Native benchmarks are a genuine advance. Efforts that write questions from real curricula, government exams and local e-learning platforms preserve cultural context, correct terminology and authentic framing, and avoid translationese entirely. Dialect-specific versions extend this further. Two limitations are worth stating plainly.

Cross-lingual comparison becomes harder: if one language's set draws on a different topic mix than another, a score difference confounds language proficiency with task difficulty. Natively sourced benchmarks are internally valid and awkward to compare across languages, which is precisely what a buyer wants to do — a problem that compounds across high-resource and low-resource languages unevenly, since low-resource sets are thinner to begin with.

Multiple choice tests retrieval, not generation. Both translated and native benchmarks overwhelmingly use multiple-choice formats, which measure knowledge retrieval and are subject to selection bias. Real-world utility depends on coherent, fluent, semantically faithful generation, and multiple choice does not assess it; researchers have been explicit that assessing multilingual generation quality remains an open problem. Native benchmarks tell a buyer whether a model handles the language and culture; generation evaluation tells them whether the output is usable. Neither substitutes for the other.

Benchmark contamination means evaluation questions appear in a model's pretraining data, inflating scores through memorisation rather than capability. Large web-crawled corpora make this likely for any benchmark that has been public for a while. Recent measurement gives a usable picture: newer benchmarks showed markedly lower contamination rates, reported as 0.0% for MILU, 1.0% for INCLUDE and 1.7% for Global MMLU, with benchmark age and prominence flagged as strong predictors of contamination risk. The corollary is uncomfortable: the most cited benchmarks are the most likely to be compromised.

Detection is also unsettled. Different methods measure different things, with surface-form overlap and memorisation-driven performance gaps producing divergent results on the same benchmark, and researchers noting that no single detection approach suffices for multilingual contamination.

Two practical responses follow. First, prefer recent benchmarks and treat long-standing leaderboard scores with more caution than their prominence suggests. Second, and more important for a company, maintain a private held-out set — an evaluation set that has never been published — as the only reliable defence against contamination, and the only test that measures a specific use case rather than a general one.

Layer 1 is native knowledge and comprehension: questions written by speakers in the language, ideally sourced from local curricula or examinations, testing whether the model knows things in that language rather than in translation. Layer 2 is generation quality: open-ended tasks judged by speakers for fluency, coherence and semantic faithfulness — the layer where multiple-choice benchmarks are silent and where users actually form their impressions. Layer 3 is task performance on the real use case: a private set drawn from real work, a company's own domain, customer phrasing and document types, uncontaminated by construction. Layer 4 is tone, register and appropriateness, judged by speakers because it is not automatable — this is the layer that determines whether output reads as respectful or rude. Layer 5 is safety and refusal behaviour, per language, since guardrails hold only where they were trained and an English-only safety evaluation tells a company almost nothing about exposure elsewhere.

Two disciplines wrap around all five layers. Report per language, never as an aggregate, because a mean is dominated by the strongest languages in the set — a point covered in more detail in how to build multilingual evaluation sets for LLMs. And define what a good answer looks like in each market before measuring, since expectations for directness, length and formality differ, and scoring everything against English norms produces confident but wrong conclusions. Programmes that also run structured preference judgements benefit from disciplined inter-annotator agreement tracking to keep raters consistent across languages.

Every layer above requires the same input: native speakers writing items, judging outputs, adjudicating disagreements and documenting decisions. There is no automated substitute, and using a model to judge output in a language it handles poorly just reproduces the original problem in a new place. That makes evaluation an operational capability rather than a research one — it needs recruitment, screening, guidelines in the target language, inter-rater agreement tracking and ongoing adjudication in each language, because models change and evaluation sets have to keep pace.

This is a large part of what Lifewood does across its 50+ languages capability: producing in-language evaluation and preference data, including tone and appropriateness judgements and per-language red-teaming, through screened native speakers working under a human-in-the-loop model across its 40+ delivery centres across 30+ countries. Companies scoping this kind of work as a managed service can review Lifewood's multilingual data collection offering and its AI data validation process for how per-language quality checks are run.

The closing point is commercial: any supplier can claim language coverage. A company that can produce per-language evaluation results, on sets authored by speakers, including generation and safety, is making a claim that can be checked. In a market where support lists are cheap, checkable evidence is the differentiator.