Short answer. Because translation converts words while leaving the underlying knowledge and assumptions unchanged. A model can answer fluently in a language and still be wrong about the food eaten at a birthday there, the appropriate level of formality, or how a request should be phrased. The measured gap is large: in the BLEnD benchmark (Myung et al., NeurIPS 2024 Datasets and Benchmarks Track), models averaged 79.22% on United States everyday knowledge asked in English and 12.18% on Ethiopian everyday knowledge asked in Amharic. That knowledge is rarely written down anywhere online, so it cannot be scraped or translated in — it has to be collected from the people who live it.
Ask a model what to bring to a colleague's house for dinner. In English it answers sensibly for a Western context. Translate that into Bengali and you get fluent Bengali advice about wine. Nothing was mistranslated. The response was simply built on knowledge from somewhere else, and the translation step faithfully carried the assumption across.
This is the failure mode that survives every quality check a non-local team can run. The grammar is correct, the terminology is consistent, the register is plausible, and the content is wrong in a way only a reader from that market can see.
Why isn't translation enough?
Because translation is a language operation applied to content that was already shaped by another culture. The words change; the worldview does not. Three categories travel badly.
| What travels badly | Examples | What a translated answer produces |
|---|---|---|
| Everyday practice | What people eat, wear, play and celebrate | Fluent advice about the wrong food, the wrong gift, the wrong occasion |
| Social norms | Formality, directness, who is addressed how, what counts as a polite refusal | Correct information delivered in a register that reads as rude or absurd |
| Reference points | Legal terms, institutions, payment methods, holidays, units | Confident references to things that do not exist in that market |
Above all three sits a harder layer still. What makes an answer helpful, appropriate or rude differs by society, and a model aligned on one society's preferences applies those preferences everywhere. That is not a knowledge gap that more facts would close; it is a calibration inherited from the alignment data.
How large is the cultural gap, measurably?
Large enough to be the dominant factor in some markets.
BLEnD was built to test exactly this: 52,600 question-and-answer pairs across 16 countries and 13 languages, including Amharic, Hausa and Sundanese, hand-crafted by native speakers rather than scraped. Building it that way matters, because a benchmark assembled from web text would measure the same online sources the models already learned from.
| Measurement | Figure |
|---|---|
| Average score, United States everyday knowledge asked in English | 79.22% |
| Average score, Ethiopian everyday knowledge asked in Amharic | 12.18% |
| Spread across cultures for the best model tested | Up to 57.34 percentage points |
| Coverage | 52,600 pairs, 16 countries, 13 languages |
Two findings underneath the headline are more useful than the headline itself.
Performance tracked how well represented a culture is online, not how difficult its language is. The ranking of cultures follows their digital footprint.
For low-resource languages, models answered better in English than in the local language. For mid-to-high-resource languages the reverse held — models did better when asked in the local language. The implication is uncomfortable and precise: for those cultures, the model knows more about the culture through English than through the language that culture actually speaks. Whatever it has absorbed came from outside descriptions rather than from the community itself.
Cultural coverage gap = Score on your reference culture in its own language − Score on the target culture in its own language
Measure it that way rather than as an overall multilingual average. An average is dominated by the well-represented cultures in the set, and it will report a model as broadly capable while it is failing completely in a specific market.
What kind of knowledge is actually missing?
The ordinary kind. The things everyone in a place knows and nobody writes down. The BLEnD authors put it directly: what people eat at birthday celebrations, the spices they cook with, the instruments young people play, the sports played at school. Common knowledge locally, uncommon in the online sources models learn from.
You cannot fix that by scraping harder or translating more. Nobody thought it needed recording, so it exists only in people.
This distinguishes cultural competence from two things it is often confused with. It is not language coverage — a model can be fluent in a language and ignorant of the culture that speaks it, which is exactly what the Amharic result shows. And it is not localisation in the production sense of adapting formats, currencies and dates. Those are surface conversions applied to content whose substance was decided elsewhere.
Where does the gap show up in a product?
| Surface | What the cultural gap looks like |
|---|---|
| Assistants and chat | Advice that is fluent, confident and inapplicable — the wine-in-Bengali failure |
| Search and recommendation | Results ranked against assumptions from a different market |
| Content generation | Copy that reads as translated even when the grammar is flawless |
| Evaluation | Green dashboards, because the test set was translated from the reference market |
| Safety and moderation | Norms enforced from one society applied to another, over- or under-blocking |
The evaluation row is the one that keeps the rest hidden. A translated benchmark carries the source culture across with it and can rank models wrongly for that market — so the measurement layer reproduces exactly the error it was installed to catch.
What closes the gap?
Native speakers producing and judging content in their own language, then verifying each other's work. There is no shortcut, because the input is lived knowledge. Four practices do most of the work.
- Author in-language; do not translate in. Prompts, answers and examples written by people from the culture, not converted from English originals. Translation can bootstrap coverage, but it cannot supply knowledge that was never in the source.
- Collect the mundane deliberately. Everyday practice is the material that is missing, so it has to be asked for explicitly. Contributors will not volunteer what they assume everyone knows; the collection instrument has to go looking for it.
- Evaluate with locally written test sets. Items authored by people from that culture, covering everyday knowledge, tone and appropriateness. A translated benchmark measures translation.
- Keep humans in the loop after launch. Cultural errors read as fluent and correct to anyone who is not from that culture — including automated checks and model-based graders, which share the assumptions that produced the error.
What to specify when commissioning this work
- Which cultures, named separately from which languages. They are not the same list, and one language may span several.
- Whether contributors live in the market now, and for how long. Diaspora knowledge drifts, particularly on everyday practice.
- How everyday-knowledge topics are elicited, and who chose the topic list.
- Whether the evaluation set is authored locally or translated, and who wrote it.
- How disagreement between local reviewers is resolved, since two people from the same market can legitimately differ.
- Consent and fair compensation for contributors — both because it is right, and because contributor networks in rare languages cannot be rebuilt once lost.
How Lifewood approaches this
This is the constraint Lifewood's delivery model is built around: collection, annotation and evaluation across 50+ languages including underrepresented dialects, produced by screened native speakers through 40+ delivery centres across 30+ countries, with human review layered over automated checks at a 95%+ accuracy threshold. The network is distributed because cultural knowledge is not portable — it has to be gathered where it lives, and a reviewer working from a translated guideline in another country cannot supply it.
See multilingual data collection, global AI data, and the companion guides on multilingual LLM training data quality and low-resource language speech data collection.
Sources and further reading
- Myung et al., "BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages", NeurIPS 2024 Datasets and Benchmarks Track (arXiv 2406.09948) — every figure quoted above.