Short answer. An English-built evaluation stack applied to another language produces a product whose non-English half scores well only because the rubric never tested it properly. MultiNRC shows the gap: over 1,000 natively written reasoning questions in French, Spanish and Chinese, on which 14 leading LLMs scored poorly — and on English equivalents of the same questions, written by native speakers, models performed around 10 percentage points better on mathematical reasoning. Existing multilingual benchmarks are typically translated from English, which biases them toward problems that were English-shaped to begin with.
Sets for LLMs?
There is a sentence in a 2026 evaluation playbook that describes the failure mode more precisely than anything else I found:
Teams shipping an English-built evaluation stack to a Hindi, Japanese or Arabic surface end up with "a bilingual-quality product: an English half that scores well and a non-English half that scores well only because the rubric cannot see what is broken."
That is the problem. Not that the model is worse in the other language, which is expected and measurable. That the evaluation apparatus is worse in the other language, so the degradation is invisible.
I have written separately in this series about how to read published multilingual benchmarks, including why translated ones align poorly with local human judgment. This piece is the companion: how to build an evaluation set yourself, which is what any company deploying in multiple markets eventually has to do.
Start with the finding that sets the bar
Before methodology, the result that should calibrate expectations.
MultiNRC, built by Scale AI, contains more than 1,000 native, linguistically and culturally grounded reasoning questions written by native speakers in French, Spanish and Chinese, across four categories: language-specific linguistic reasoning, wordplay and riddles, cultural and tradition reasoning, and mathematical reasoning with cultural relevance.
They evaluated 14 leading LLMs covering most model families. None scored above 50%.
And because the team also produced English equivalents of the culturally grounded questions through manual translation by native speakers fluent in English, they could isolate the language effect directly. Most models performed substantially better on mathematical reasoning in English than in the original languages, by around 10 percentage points, on the same underlying questions.
The critique underneath that design is the important part: existing multilingual reasoning benchmarks are typically constructed by translating English benchmarks, which biases them toward reasoning problems with context in English language and culture. A translated test measures whether the model can handle a translated English problem. It does not measure whether the model can reason in that language about that culture.
The three construction modalities
Published practice uses three approaches, and most serious benchmarks combine them.
Machine translation with expert post-editing. Large-scale benchmarks including MuBench, BenchMAX and MMLU-ProX use machine and LLM-assisted translation pipelines with expert post-editing to preserve semantic, terminological and cultural fidelity. The advantage is parallel data construction, which enables direct cross-lingual comparison because every language sees the same item.
Hybrid human review. Translation followed by structured multi-annotator adjudication.
Native authoring. Questions written in the target language by native speakers, as in MultiNRC. Highest authenticity, no parallelism, highest cost.
The trade is genuine and worth stating for anyone scoping this. Parallel translated sets let you compare across languages. Natively authored sets tell you whether the model works in that language. They answer different questions and a mature programme needs both.
What construction actually costs
Two published figures give a usable scoping anchor, which is rare in this area.
BenchMAX extended evaluation across 16 non-English languages covering six core LLM capabilities, using a threestep pipeline: translate from English, post-edit each sample by three human annotators, then select the final translation version.
Three annotators per item, across sixteen languages, is the cost driver in that design.
MIDB reports the effort explicitly for extending AlpacaEval and MT-Bench, both originally English-only, to 16 languages: a team of 20 professional translators dedicating a total of 175 person-days to test set development.
That is the number to put in a project plan. Roughly 11 person-days per language for extending two existing benchmarks, before any native authoring.
The QA methodology that holds up
The most transparent published quality process I found comes from a multilingual intent classification benchmark, and it is worth copying almost exactly.
Their pipeline: candidates pre-screened with LLM-assisted filtering and consistency checks, then reviewed by two annotators for each language. A 5% subset double-annotated for test set construction, with annotator agreement on that subset at 95%. Remaining disagreements adjudicated by a third expert, and samples without consensus removed entirely.
Three things in that design are worth naming.
Automated filtering runs first, as triage rather than judgement, which is the same architecture that works everywhere else in data operations.
Agreement is measured on a defined subset, not assumed. And the 95% figure is reported, which lets a reader judge the set's reliability rather than trusting it.
No-consensus items are removed rather than force-resolved. This is the discipline most teams skip. An item two qualified annotators cannot agree on is not a hard item, it is an ambiguous one, and including it adds noise to every model's score.
The same benchmark makes a distinction worth adopting in scoping language: native evaluation versus synthetic evaluation. Their native evaluation used a 674-example human-verified subset of held-out real multilingual traffic, with cross-language semantic duplicates removed.
Six hundred and seventy-four examples. Held out from real traffic. Deduplicated across languages. That is a more useful evaluation asset than tens of thousands of translated items, and it is achievable for most companies.
Stratification, and the mistakes it prevents
The practical guidance here is unusually specific and worth quoting close to the original.
Stratify the golden set by language and by script. And two explicit warnings: do not lump all CJK together, and do not assume Spanish and Portuguese behave the same.
Both errors are common because they are administratively convenient. Chinese, Japanese and Korean share script characteristics and almost nothing else relevant to model behaviour. Spanish and Portuguese are close enough that teams routinely treat one as a proxy for the other, and close enough that the failures are subtle rather than obvious.
The associated quality bar: hire native annotators per language and enforce kappa above 0.7 per language.
Per language. Not an aggregate agreement figure across the whole set, which can look healthy while one language is at 0.4.
This is the same per-language reporting discipline I have argued for across this series, applied to evaluation construction rather than to production data.
The four rubrics worth shipping by default
This is the most immediately actionable content I found, and I would recommend it to any team building multilingual evaluation. Four native rubrics, each catching a failure that a general quality score misses:
IdiomTransferQuality. Did the translation preserve idiomatic intent rather than render literally? Catches the output that is technically accurate and reads as machine-produced.
CulturalRegisterAdherence. Did the response match the cultural register expected for the user's locale? Catches the answer that is correct and inappropriately casual, or correct and stiffly formal, for that market.
RefusalPreservation. Did a refusal in the source language remain a refusal in the target language? This is a safety rubric, and it is the one I would prioritise. Guardrails are trained per language, and a model that declines a request in English and complies in another language is a security failure, not a quality one.
FormalityCorrectness. Did the response hit the expected formality level? Particularly important in languages with grammaticalised politeness systems, where getting this wrong is not a stylistic miss but a social error.
None of these can be evaluated by an English-tuned judge model, which is the structural reason native annotators are required rather than preferred.
The judge problem
A note on automated adjudication, because it is where cost pressure pushes and where it should be resisted selectively.
Published benchmarks do use LLM adjudicators: GPT-4 in OMGEval, GPT-4o in MMLU-ProX, and similar approaches elsewhere. Combined with multi-stage human annotation, this is a reasonable architecture for scale.
The limitation is that judge models inherit the same language-proficiency curve as the models being judged. A judge that is strong in English and weak in Yoruba will assess Yoruba output unreliably, and it will do so with the same confident scoring behaviour it uses in English. Its errors will not look like errors.
The practical position: automated judging for triage, throughput and regression testing across a large set; human native judgement for the calibration subset, for the four rubrics above, and for any language where the judge model's own competence is uncertain, which is most of them.
Where we come at this from
Declaring the interest: Lifewood builds evaluation and preference data across 50-plus languages through native speakers in delivery centres in more than 30 countries, so evaluation set construction is part of what we do.
Two things I would say to anyone scoping this work.
The first is that the highest-value asset is a private native set, not a large translated one. The logistics benchmark's 674 human-verified native examples from held-out real traffic is a better evaluation instrument than a machine-translated set fifty times its size, because it measures your deployment rather than a benchmark's proxy for it, and because it cannot be contaminated by appearing in training data.
The second is that the annotator requirement is the binding constraint, and it is a recruitment problem before it is a methodology problem. A kappa above 0.7 per language requires two or three qualified native annotators per language, available concurrently, working to a shared guideline. For sixteen languages that is thirty to fifty people with the right linguistic profile, which is a delivery network question rather than a research design question. This is where projects slip, and it is worth planning against the recruitment timeline rather than the annotation timeline.
A build checklist
Decide what each set is for. Parallel translated for cross-language comparison, native authored for whether the model works there. Do not conflate them.
Budget realistically. Roughly 11 person-days per language to extend existing English benchmarks, per the published MIDB figure, before native authoring.
Run LLM-assisted filtering as triage, then human review, in that order.
Use two annotators per language, with a double-annotated subset to measure agreement, a third expert for adjudication, and removal rather than forced resolution for no-consensus items.
Enforce kappa above 0.7 per language, reported per language.
Stratify by language and script. Not by region, not by script family.
Ship the four native rubrics, prioritising refusal preservation.
Build a private held-out native set from real traffic where you have it, with cross-language semantic duplicates removed.
Use automated judges for throughput, not for the calibration set.
Key takeaways
- An English-built evaluation stack applied to another language produces a product where the non-English half scores well only because the rubric cannot see what is broken.
- MultiNRC contains over 1,000 natively written reasoning questions in French, Spanish and Chinese across four categories. Of 14 leading LLMs evaluated, none scored above 50%.
- On the same questions with English equivalents produced by native speakers, models performed around 10 percentage points better on mathematical reasoning in English.
- Existing multilingual reasoning benchmarks are typically translated from English, biasing them toward problems with English-language and English-cultural context.
- Three construction modalities: machine translation with expert post-editing, hybrid human review, and native authoring. Parallel sets enable cross-language comparison; native sets measure whether the model works in that language.
- BenchMAX covered 16 non-English languages and six capabilities via translation, post-editing by three human annotators per sample, and final version selection.
- MIDB reports 20 professional translators and 175 person-days to extend AlpacaEval and MT-Bench to 16 languages, roughly 11 person-days per language.
- A published QA pipeline used LLM-assisted filtering, two annotators per language, a 5% double-annotated subset with 95% agreement, third-expert adjudication, and removal of no-consensus samples.
- Their native evaluation set was 674 human-verified examples from held-out real multilingual traffic with cross- language semantic duplicates removed.
- Stratify golden sets by language and script. Do not lump CJK together and do not treat Spanish and Portuguese as equivalent.
- Enforce inter-annotator kappa above 0.7 per language, reported per language rather than in aggregate.
- Four native rubrics worth shipping by default: IdiomTransferQuality, CulturalRegisterAdherence, RefusalPreservation and FormalityCorrectness.
- Refusal preservation is a safety rubric: a model that refuses in English and complies in another language is a security failure.
- Published benchmarks use LLM adjudicators including GPT-4 and GPT-4o, but judge models inherit the same language proficiency curve as the models they assess, so human native judgement is required for calibration subsets.
- The binding constraint is annotator recruitment: kappa above 0.7 across sixteen languages requires thirty to fifty qualified native annotators working concurrently.
Sources and further reading
- "MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs", Scale AI, arXiv, on the 1,000-plus native questions, four reasoning categories, 14 models evaluated with none above 50%, and the 10point English advantage on mathematical reasoning
- Future AGI, "Multilingual LLM Evaluation: A 2026 Playbook for Non-English", on the bilingual-quality product framing, stratification by language and script, the kappa 0.7 per language bar, and the four native rubrics
- "BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models", arXiv, on the 16-language six-capability scope and the three-step translate, post-edit by three annotators, and select pipeline
- "MIDB: Multilingual Instruction Data Booster", arXiv, on the 20 translators and 175 person-days required to extend.
- HAlpacaEval and MT-Bench to 16 languages
- "From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service", arXiv, on LLM-assisted filtering, two annotators per language, the 5% double-annotated subset at 95% agreement, thirdexpert adjudication, removal of no-consensus items, and the 674-example native evaluation subset