Short answer. Multilingual text data enters an LLM at four distinct stages, and each needs a different kind of data: pretraining needs volume above a per-language token floor, the tokenizer needs script diversity, instruction tuning needs prompts and answers written by native speakers, and evaluation needs test sets built in-language. Recent research has also overturned the old assumption that adding languages necessarily costs performance. The penalty comes from thin token counts and low-quality corpora, not from language count — which turns a modelling trade-off into a data-operations problem.
Treating "multilingual data" as one procurement item is the most common planning error in this work. It leads teams to over-invest in the cheapest stage and under-invest in the two that decide whether the model is actually usable in a language.
This piece separates the four stages, then works through what the 2025 and 2026 research says about token floors, quality filtering and why instruction data still has to be written by people.
Where does multilingual text data actually enter LLM training?
| Stage | What it is | What the currency is | Can it be bulk-sourced? |
|---|---|---|---|
| Pretraining corpus | Enormous volumes of raw, mostly unlabelled text where the model learns the shape of a language | Tokens per language | Yes |
| Tokenizer training | Usually a sample drawn from the pretraining corpus | Script and language diversity in the sample | Yes, but the composition matters disproportionately |
| Instruction and preference data | Prompts and responses that teach the model to be useful, follow instructions and adopt an appropriate register | Authorship quality | No |
| Evaluation sets | Test data that reveals whether the previous three worked | Independence from the training set | No |
The requirements pull in different directions. Pretraining rewards scale. Instruction tuning rewards authorship. Evaluation rewards independence. A programme that treats all of it as "get more text in these languages" will spend most of its budget on the cheapest stage.
The tokenizer decision deserves separate attention because it is small, cheap and permanent. A tokenizer built on an English-dominant sample encodes every other language inefficiently for the life of the model, which shows up as cost, latency and reduced effective context in every language you ship.
Is the curse of multilinguality real?
It is measurable, but recent work suggests it has been widely misdiagnosed.
The curse of multilinguality was described in 2020: under a fixed model capacity, adding languages first helps — especially low-resource ones — then starts to hurt both monolingual and cross-lingual performance. For years it was read as a hard trade-off. Coverage or quality, pick one.
A 2025 study revisited this at proper scale, training 1.1B and 3B parameter models on corpora ranging from 25 to 400 languages. Its findings complicate the old story considerably:
- Combining English and multilingual data did not necessarily degrade performance for either group, provided each language had a sufficient number of tokens in the corpus.
- Using English as a pivot language produced benefits across language families — and, contrary to expectation, choosing a pivot from within a language's own family was not necessarily better.
- The authors attribute the curse to the finite capacity of the model and to data distributions that amplify the influence of languages represented by poor-quality data, rather than to adding languages as such.
That is a meaningfully different problem. "Languages compete for capacity" implies you should cut languages. "Thin, low-quality corpora drag on everything" implies you should fix the corpora — and that is a data-operations problem, which is solvable.
How much text does one language need?
Enough to clear a floor, and the floor matters more than the share. Below it, a language contributes noise; above it, it contributes capability.
The practical implication is that per-language token volume is the variable to watch, not the number of languages on the list. A model trained on 400 languages where most have negligible token counts is not a 400-language model. It is a model with 400 labels and a handful of functioning languages.
This is also why proportional sampling from the natural web distribution fails so badly. Sample languages in proportion to how much text exists online and English takes roughly half while the tail gets almost nothing — exactly reproducing the imbalance you were trying to correct. The standard countermeasure is temperature sampling, which flattens the distribution by upweighting smaller languages, but pushed too far it causes low-resource data to be repeated until the model overfits to it.
Work on multilingual scaling laws has made this more tractable by deriving optimal sampling ratios that minimise total loss across languages. Encouragingly for anyone on a budget, ratios derived from small models of around 85M parameters were found to generalise to models several orders of magnitude larger — so a team can search for the right mixture cheaply and apply it at scale.
The uncomfortable conclusion for low-resource languages is that clearing the floor often requires text that does not exist online yet. At that point the mixture question becomes a collection question.
Does data quality beat data quantity?
Decisively, in the multilingual setting. This is the most encouraging result in recent multilingual pretraining research.
Work on model-based data selection applied quality filtering across diverse language families and scripts, then trained 1B parameter models to compare. The filtered data matched the baseline MMLU score using as little as 15% of the training tokens.
The second finding is more striking. When a multilingual model was compared against monolingual counterparts trained on the same number of tokens in the language of interest, the multilingual model trained on filtered data outperformed its monolingual equivalent. On unfiltered data, it suffered the expected penalty. Same architecture, same token budget, opposite outcome — decided entirely by corpus quality.
For anyone planning a multilingual programme, this reframes the budget question. The instinct is to ask how much more text can be acquired. The better question is how much of the existing text is worth training on, and what verified in-language material could replace the rest. Filtering and curation are not overheads on top of collection; in multilingual training they are where much of the performance comes from.
Why does instruction data have to be written by people?
Because it teaches behaviour rather than language, and behaviour is culturally specific.
The clearest evidence comes from the Aya project, still the reference point for multilingual instruction tuning. It produced two things of very different character:
| Aya Dataset | Aya Collection | |
|---|---|---|
| Size | ~204,000 prompt and completion pairs | ~513 million instances |
| Languages | 65 | 114 |
| How it was built | Written and reviewed by fluent speakers; ~3,000 collaborators in 119 countries | Largely templating and machine-translating existing English datasets |
| What it delivers | Behaviour: register, appropriateness, engagement | Breadth |
Both were necessary. Translation and templating deliver breadth no human effort could match at that cost. But the researchers were direct about why the smaller set mattered: open-ended instruction data from human annotators is difficult and expensive to obtain, and it is what makes a model engaging and appropriate in conversation rather than merely correct.
There is a structural point underneath. Pretraining teaches a model the language. Instruction tuning teaches it how to behave in that language, and every culture answers that differently — what counts as a polite refusal, an appropriate level of directness, a culturally sensible example. You cannot translate your way to it.
How do you know the multilingual data worked?
Only by evaluating per language, on test sets built by speakers of that language. Three failure modes recur.
- Averaging across languages. A single multilingual score is dominated by the high-resource languages in the set. A model can look solid overall while being unusable in a third of its supported languages.
- Translated benchmarks. Translating an English test set produces a test of translated English, not of the language. It rewards models that think in English and answer in translation — the exact behaviour you were trying to eliminate.
- Testing only what is easy to measure. Accuracy is straightforward. Register, tone, cultural appropriateness and refusal behaviour are not, and they are what users actually notice.
The Aya work is instructive here too: alongside the model, the team built evaluation suites spanning almost a hundred languages, including human evaluation rather than automated scoring alone.
How Lifewood approaches this
Lifewood's work sits at the two stages that cannot be bought in bulk: in-language instruction, preference and evaluation data, produced by trained native speakers rather than translated in. Prompts, responses, preference ranking and evaluation sets are authored in the language, which is the only way the behaviour layer acquires the register and cultural framing the research says translation cannot carry.
Quality is verified under a human-in-the-loop model against a customer-approved gold set at a 95%+ accuracy SLA, and reported per language rather than as an aggregate, because an averaged figure is dominated by the largest languages in the set and hides the ones that need attention.
For languages where clearing the token floor means collecting text that does not exist online, the constraint is people in the right places rather than tooling. 50+ languages including underrepresented dialects, 40+ delivery centres across 30+ countries and 56,788 registered contributors are what make that a delivery plan rather than an aspiration. See horizontal vs vertical LLM training data and multilingual LLM training data quality.
Sources and further reading
- Revisiting Multilingual Data Mixtures in Language Model Pretraining (2025), arXiv — the 25-to-400-language study and the reinterpretation of the curse of multilinguality.
- Conneau et al. on the curse of multilinguality, discussed in the above.
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection, arXiv — the 15%-of-tokens quality filtering result.
- Scaling Laws for Multilingual Language Models, arXiv — optimal sampling ratios and the 85M-parameter transfer finding.
- Singh et al., Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning, arXiv.
- Üstün et al., Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model, Cohere.