How to Build Multilingual Evaluation Sets for LLMs

There is a sentence in a 2026 evaluation playbook that describes the failure mode more precisely than anything else on the topic: teams shipping an English-built evaluation stack to a Hindi, Japanese or Arabic surface end up with a bilingual-quality product — an English half that scores well and a non-English half that scores well only because the rubric cannot see what is broken. That is the problem. Not that the model is worse in the other language, which is expected and measurable. That the evaluation apparatus itself is worse in the other language, so the degradation is invisible. This piece sets out how to build an evaluation set from scratch, which is what any company deploying in multiple markets eventually has to do; a companion piece in this series covers how to read published multilingual benchmarks and why translated ones align poorly with local human judgment.

The questions span four categories: language-specific linguistic reasoning, wordplay and riddles, cultural and tradition reasoning, and mathematical reasoning with cultural relevance. Scale AI evaluated 14 leading LLMs covering most model families, and none scored above 50%. Because the team also produced English equivalents of the culturally grounded questions through manual translation by native speakers fluent in English, they could isolate the language effect directly: most models performed substantially better on mathematical reasoning in English than in the original languages, by around 10 percentage points, on the same underlying questions.

The critique underneath that design is the important part. Existing multilingual reasoning benchmarks are typically constructed by translating English benchmarks, which biases them toward reasoning problems with context in English language and culture. A translated test measures whether the model can handle a translated English problem. It does not measure whether the model can reason in that language about that culture.

Published practice uses three construction approaches, and most serious benchmarks combine them.

• Machine translation with expert post-editing. Large-scale benchmarks including MuBench, BenchMAX and MMLU-ProX use machine and LLM-assisted translation pipelines with expert post-editing to preserve semantic, terminological and cultural fidelity. The advantage is parallel data construction, which enables direct cross-lingual comparison because every language sees the same item.
• Hybrid human review. Translation followed by structured multi-annotator adjudication.
• Native authoring. Questions written in the target language by native speakers, as in MultiNRC. Highest authenticity, no parallelism, highest cost.

The trade is genuine and worth stating for anyone scoping this work: parallel translated sets let you compare across languages, natively authored sets tell you whether the model works in that language, and they answer different questions.

BenchMAX extended evaluation across 16 non-English languages covering six core LLM capabilities, using a three-step pipeline: translate from English, post-edit each sample by three human annotators, then select the final translation version. Three annotators per item, across sixteen languages, is the cost driver in that design.

MIDB reports the effort explicitly for extending AlpacaEval and MT-Bench, both originally English-only, to 16 languages: a team of 20 professional translators dedicating a total of 175 person-days to test set development. That works out to roughly 11 person-days per language for extending two existing benchmarks, before any native authoring.

Their pipeline runs candidates through LLM-assisted filtering and consistency checks first, then two annotators review each item for every language. A 5% subset is double-annotated for test set construction, with annotator agreement on that subset reported at 95%. Remaining disagreements are adjudicated by a third expert, and samples without consensus are removed entirely rather than force-resolved.

Three things in that design are worth naming. Automated filtering runs first, as triage rather than judgement, which is the same architecture that works everywhere else in data operations. Agreement is measured on a defined subset, not assumed, and the figure is reported so a reader can judge the set's reliability. No-consensus items are removed rather than force-resolved: an item two qualified annotators cannot agree on is not a hard item, it is an ambiguous one, and including it adds noise to every model's score.

The same benchmark distinguishes native evaluation from synthetic evaluation. Its native evaluation used a 674-example human-verified subset of held-out real multilingual traffic, with cross-language semantic duplicates removed — a smaller but more useful evaluation asset than tens of thousands of translated items, and achievable for most companies.

Both errors are administratively convenient, which is why they persist. Chinese, Japanese and Korean share script characteristics and almost nothing else relevant to model behaviour. Spanish and Portuguese are close enough that teams routinely treat one as a proxy for the other, and close enough that the resulting failures are subtle rather than obvious.

The associated quality bar is to hire native annotators per language and enforce kappa — a statistical measure of inter-annotator agreement that corrects for chance — above 0.7 per language, reported per language rather than as one aggregate figure across the whole set, which can look healthy while one language sits at 0.4. For more on how to read that statistic, see inter-annotator agreement measures.

• IdiomTransferQuality. Did the translation preserve idiomatic intent rather than render literally? Catches output that is technically accurate and reads as machine-produced.
• CulturalRegisterAdherence. Did the response match the cultural register expected for the user's locale? Catches an answer that is correct and inappropriately casual, or correct and stiffly formal, for that market.
• RefusalPreservation. Did a refusal in the source language remain a refusal in the target language? This is a safety rubric and the one to prioritise: guardrails are trained per language, and a model that declines a request in English and complies in another language is a security failure, not a quality one.
• FormalityCorrectness. Did the response hit the expected formality level? Particularly important in languages with grammaticalised politeness systems, where getting this wrong is a social error, not a stylistic miss.

Published benchmarks do use LLM adjudicators — GPT-4 in OMGEval, GPT-4o in MMLU-ProX, and similar approaches elsewhere — combined with multi-stage human annotation, which is a reasonable architecture for scale. The limitation is that a judge that is strong in English and weak in Yoruba will assess Yoruba output unreliably, and it will do so with the same confident scoring behaviour it uses in English: its errors will not look like errors. See how reliable an LLM judge actually is for the broader case. The practical position is automated judging for triage and regression testing across a large set, and human native judgement for the calibration subset, for the four rubrics above, and for any language where the judge model's own competence is uncertain — which is most of them.

Two things follow from doing this work at scale. First, the highest-value asset is a private native set, not a large translated one: a held-out, human-verified native sample from real traffic is a better evaluation instrument than a machine-translated set fifty times its size, because it measures an actual deployment rather than a benchmark's proxy for it, and it cannot be contaminated by appearing in training data. A comparable pattern shows up in how multilingual data collection programmes are scoped more broadly.

Second, the annotator requirement is the binding constraint, and it is a recruitment problem before it is a methodology problem. A kappa above 0.7 per language typically needs two or three qualified native annotators per language, available concurrently, working to a shared guideline; for sixteen languages that is thirty to fifty people with the right linguistic profile, which is a delivery-network question rather than a research-design question. This is where projects slip, and it is worth planning against the recruitment timeline rather than the annotation timeline — the same constraint that shapes gold sets and audit sampling more generally, and one reason buyers compare providers on multilingual AI training data coverage before committing to a build.

• Decide what each set is for: parallel translated sets for cross-language comparison, native-authored sets for whether the model works there. Do not conflate them.
• Budget realistically: roughly 11 person-days per language to extend existing English benchmarks, before native authoring.
• Run LLM-assisted filtering as triage, then human review, in that order.
• Use two annotators per language, with a double-annotated subset to measure agreement, a third expert for adjudication, and removal rather than forced resolution for no-consensus items.
• Enforce kappa above 0.7 per language, reported per language.
• Stratify by language and script, not by region and not by script family.
• Ship the four native rubrics, prioritising refusal preservation.
• Build a private held-out native set from real traffic where you have it, with cross-language semantic duplicates removed.
• Use automated judges for throughput, not for the calibration set.