Short answer. Multilingual text data enters an LLM at four distinct stages, and each needs a different kind of data: pretraining needs volume above a per-language token floor, the tokenizer needs script diversity, instruction tuning needs prompts and answers written by native speakers, and evaluation needs test sets built in-language. Recent research has also overturned the old assumption that adding languages necessarily costs performance. The penalty comes from thin token counts and low-quality corpora, not from language count — which turns a modelling trade-off into a data-operations problem.
Treating "multilingual data" as one procurement item is the most common planning error in this work. It leads teams to over-invest in the cheapest stage and under-invest in the two that decide whether the model is actually usable in a language.
This piece separates the four stages, then works through what the 2025 research says about token floors, quality filtering and why instruction data still has to be written by people.
Key takeaways
- Multilingual text data serves four separate stages of LLM training — pretraining, tokenizer construction, instruction/preference tuning and evaluation — and each rewards a different property of the data.
- A 2025 study training 1.1B and 3B parameter models on 25 to 400 languages found that adding languages does not necessarily hurt performance once each language clears a sufficient token count.
- Quality filtering has matched baseline benchmark scores using as little as 15% of the training tokens in multilingual pretraining.
- Instruction and preference data has to be written by native speakers because it teaches culturally specific behaviour, not just language; the Aya project paired 513 million translated instances with 204,000 human-written pairs rather than relying on translation alone.
- Multilingual models should be evaluated per language on test sets authored by speakers of that language, because averaged scores and translated benchmarks both hide failures that matter.
Where does multilingual text data actually enter LLM training?
Multilingual text data enters model training through four distinct stages, and each one needs a different kind of data.
The pretraining corpus is the enormous volume of raw, mostly unlabelled text from which the model learns the shape of a language; its currency is tokens per language, and it can be bulk-sourced. Tokenizer training usually draws a sample from that same corpus, and here script and language diversity in the sample matters disproportionately even though the sample can also be bulk-sourced. Instruction and preference data — the prompts and responses that teach a model to be useful, follow instructions and adopt an appropriate register — cannot be bulk-sourced; its currency is authorship quality. Evaluation sets, which reveal whether the previous three stages worked, depend on independence from the training set and likewise cannot be bulk-sourced.
| Stage | What it is | What the currency is | Can it be bulk-sourced? |
|---|---|---|---|
| Pretraining corpus | Enormous volumes of raw, mostly unlabelled text where the model learns the shape of a language | Tokens per language | Yes |
| Tokenizer training | Usually a sample drawn from the pretraining corpus | Script and language diversity in the sample | Yes, but the composition matters disproportionately |
| Instruction and preference data | Prompts and responses that teach the model to be useful, follow instructions and adopt an appropriate register | Authorship quality | No |
| Evaluation sets | Test data that reveals whether the previous three worked | Independence from the training set | No |
The requirements pull in different directions. Pretraining rewards scale. Instruction tuning rewards authorship. Evaluation rewards independence. A programme that treats all of it as "get more text in these languages" will spend most of its budget on the cheapest stage.
The tokenizer decision deserves separate attention because it is small, cheap and permanent. A tokenizer built on an English-dominant sample encodes every other language inefficiently for the life of the model, which shows up as cost, latency and reduced effective context in every language shipped. For a broader view of how these stages compare across a training programme, see horizontal vs vertical LLM training data.
Is the curse of multilinguality real?
It is measurable, but recent work suggests it has been widely misdiagnosed.
The curse of multilinguality is the finding, first described in 2020, that under a fixed model capacity, adding languages first helps — especially low-resource ones — then starts to hurt both monolingual and cross-lingual performance. For years it was read as a hard trade-off: coverage or quality, pick one.
A 2025 study revisited this at proper scale, training 1.1B and 3B parameter models on corpora ranging from 25 to 400 languages. Its findings complicate the old story considerably:
- Combining English and multilingual data did not necessarily degrade performance for either group, provided each language had a sufficient number of tokens in the corpus.
- Using English as a pivot language produced benefits across language families — and, contrary to expectation, choosing a pivot from within a language's own family was not necessarily better.
- The authors attribute the curse to the finite capacity of the model and to data distributions that amplify the influence of languages represented by poor-quality data, rather than to adding languages as such.
That is a meaningfully different problem. "Languages compete for capacity" implies cutting languages. "Thin, low-quality corpora drag on everything" implies fixing the corpora instead — and that is a data-operations problem, which is solvable. Teams weighing which languages to prioritise first can see how that trade-off plays out in practice in high-resource vs low-resource languages in AI training.
How much text does one language need?
Enough to clear a floor, and the floor matters more than the share. Below it, a language contributes noise; above it, it contributes capability.
The practical implication is that per-language token volume is the variable to watch, not the number of languages on the list. A model trained on 400 languages where most have negligible token counts is not a 400-language model. It is a model with 400 labels and a handful of functioning languages.
This is also why proportional sampling from the natural web distribution fails so badly. Sample languages in proportion to how much text exists online and English takes roughly half while the tail gets almost nothing — exactly reproducing the imbalance the sampling was meant to correct. The standard countermeasure is temperature sampling, which flattens the distribution by upweighting smaller languages, but pushed too far it causes low-resource data to be repeated until the model overfits to it.
Work on multilingual scaling laws has made this more tractable by deriving optimal sampling ratios that minimise total loss across languages. Encouragingly for anyone on a budget, ratios derived from small models of around 85M parameters were found to generalise to models several orders of magnitude larger — so a team can search for the right mixture cheaply and apply it at scale.
The uncomfortable conclusion for low-resource languages is that clearing the floor often requires text that does not exist online yet. At that point the mixture question becomes a collection question — the subject of how to scope language coverage at locale level.
Does data quality beat data quantity?
Decisively, in the multilingual setting. This is the most encouraging result in recent multilingual pretraining research.
Work on model-based data selection applied quality filtering across diverse language families and scripts, then trained 1B parameter models to compare. The filtered data matched the baseline benchmark score using as little as 15% of the training tokens.
The second finding is more striking. When a multilingual model was compared against monolingual counterparts trained on the same number of tokens in the language of interest, the multilingual model trained on filtered data outperformed its monolingual equivalent. On unfiltered data, it suffered the expected penalty. Same architecture, same token budget, opposite outcome — decided entirely by corpus quality.
For anyone planning a multilingual programme, this reframes the budget question. The instinct is to ask how much more text can be acquired. The better question is how much of the existing text is worth training on, and what verified in-language material could replace the rest. Filtering and curation are not overheads on top of collection; in multilingual training they are where much of the performance comes from, which is why multilingual LLM training data quality is worth treating as its own workstream.
Why does instruction data have to be written by people?
Because it teaches behaviour rather than language, and behaviour is culturally specific.
Instruction tuning is the training stage that teaches a pretrained model to follow prompts and hold a register appropriate to the conversation, as distinct from pretraining, which only teaches it the shape of a language. The clearest evidence for why this stage resists automation comes from the Aya project, still the reference point for multilingual instruction tuning. It produced two things of very different character:
| Aya Dataset | Aya Collection | |
|---|---|---|
| Size | ~204,000 prompt and completion pairs | ~513 million instances |
| Languages | 65 | 114 |
| How it was built | Written and reviewed by fluent speakers; ~3,000 collaborators in 119 countries | Largely templating and machine-translating existing English datasets |
| What it delivers | Behaviour: register, appropriateness, engagement | Breadth |
Both were necessary. Translation and templating deliver breadth no human effort could match at that cost. But the researchers were direct about why the smaller, human-written set mattered: open-ended instruction data from human annotators is difficult and expensive to obtain, and it is what makes a model engaging and appropriate in conversation rather than merely correct — a distinction covered further in beyond translation: why AI needs culturally relevant data.
There is a structural point underneath. Pretraining teaches a model the language. Instruction tuning teaches it how to behave in that language, and every culture answers that differently — what counts as a polite refusal, an appropriate level of directness, a culturally sensible example. Translation alone cannot produce it.
How do you know the multilingual data worked?
Only by evaluating per language, on test sets built by speakers of that language. Three failure modes recur.
- Averaging across languages. A single multilingual score is dominated by the high-resource languages in the set. A model can look solid overall while being unusable in a third of its supported languages.
- Translated benchmarks. Translating an English test set produces a test of translated English, not of the language. It rewards models that think in English and answer in translation — the exact behaviour the evaluation was meant to catch.
- Testing only what is easy to measure. Accuracy is straightforward. Register, tone, cultural appropriateness and refusal behaviour are not, and they are what users actually notice.
The Aya work is instructive here too: alongside the model, the team built evaluation suites spanning almost a hundred languages, including human evaluation rather than automated scoring alone. Teams building their own test sets from scratch can start from how to build multilingual evaluation sets for LLMs.
How does Lifewood approach multilingual text data?
Lifewood works at the two stages that cannot be bought in bulk: in-language instruction, preference and evaluation data, produced by trained native speakers rather than translated in.
Prompts, responses, preference ranking and evaluation sets are authored in the language, which is the only way the behaviour layer acquires the register and cultural framing that translation cannot carry. Quality is verified under a human-in-the-loop model against a customer-approved gold set at a 95%+ accuracy SLA, and reported per language rather than as an aggregate, because an averaged figure is dominated by the largest languages in the set and hides the ones that need attention — the same pitfall described above for evaluation.
For languages where clearing the token floor means collecting text that does not exist online, the constraint is people in the right places rather than tooling. 100+ languages including underrepresented dialects, 40+ delivery centres across 30+ countries and 56,000+ registered contributors are what make that a delivery plan rather than an aspiration. Programmes that need this scale end-to-end can see how the operation is structured via managed multilingual data collection and enterprise LLM training data.