Short answer. Yes, and in multilingual work it frequently does. A manual audit of 205 web-crawled language corpora found at least 15 containing no usable text at all and 87 falling below 50% usable content. Adding data of that standard teaches a model the wrong language, the wrong facts or nothing at all, while consuming the training budget that verified data would have used. Below a quality threshold, volume stops helping and starts costing.
Key takeaways
- More data hurts when it crowds out good data, teaches false associations or amplifies duplicated content, and the effect is strongest in low-resource languages.
- An audit of 205 web-crawled corpora found at least 15 with no usable text, 87 below 50% usable, and 82 mislabelled or using ambiguous language codes.
- 44 of 65 audited CCAligned languages and 19 of 20 WikiMatrix languages fell under 50% correct sentences.
- Noise takes four recurring forms: duplication, wrong language, misalignment in parallel data, and non-language content, and each needs a different fix.
- Filtered multilingual corpora have matched baseline benchmark results on as little as 15% of the tokens, so filtering is a performance lever, not just cleanup.
How can adding data make a model worse?
Through three mechanisms, all of which get stronger as a language gets smaller.
It crowds out good data. Training budgets are finite, so every token spent on a garbled document is one not spent on a correct one, and in a low-resource language correct documents are the scarce asset. Usable text, in this context, means a sentence or document a fluent speaker would recognise as correct, on-topic content in the language it claims to be.
It teaches the wrong thing. A corpus labelled as one language but containing another does not simply fail to help; it trains a false association between a language code and content that does not belong to it.
It amplifies whatever dominates. Duplicated content is learned disproportionately, so boilerplate and scraped repetitions become the model's idea of the language.
The asymmetry matters. In English, noise is diluted by enormous volumes of clean text. In a language with a few hundred thousand usable sentences, a noisy corpus can be most of what the model sees. Teams building multilingual evaluation sets run into the same asymmetry: a benchmark drawn from noisy source text measures noise, not model quality.
How bad is web-crawled multilingual data really?
Worse than most teams assume, and the evidence has been public since 2022.
Kreutzer and colleagues manually audited 205 language-specific corpora — collections of text gathered for a single language and released for others to train on — from five major public datasets: CCAligned, ParaCrawl, WikiMatrix, OSCAR and mC4. The findings matter because these datasets underpin a great deal of multilingual model training, including the work behind high-resource vs low-resource language comparisons.
At least 15 corpora contained no usable text at all, meaning not a single correct sentence in the audited sample. 87 languages fell below 50% usable data. In CCAligned, 44 of the 65 audited languages had under 50% correct sentences, and in WikiMatrix 19 of 20 did. For WikiMatrix, roughly two-thirds of audited samples were misaligned on average, with sentence pairs that looked structurally similar while describing different facts.
Separately, 82 corpora were mislabelled or used nonstandard or ambiguous language codes, meaning the dataset was not reliably about the language it claimed.
The authors added a point that should be encouraging: these problems are easy to detect, even for people who do not speak the language fluently. Nobody had checked.
What does the noise actually consist of?
Four recurring categories, each requiring a different fix.
Duplication. Redundancy in crawled data is extensive: work on Indic corpora reports it across roughly 70% of crawled pages, and one Portuguese pipeline removed around 40% of remaining pages by deduplicating — removing near-identical repeated text — within a single crawl. This is the same problem addressed in guidance on how to clean and deduplicate a pretraining corpus.
Wrong or mislabelled language. Content in a different language, in a romanised variant of the claimed one, or under an ambiguous code. This is the failure that makes a dataset actively misleading rather than merely thin.
Misalignment. In parallel corpora, sentence pairs that are not translations of each other. A model trained on these learns false equivalences with confidence.
Non-language content. Boilerplate, navigation, autogenerated text, code fragments and spam, all present at material rates in the audited low-resource corpora. Measuring how much of a corpus is genuinely diverse, useful content is part of what dataset diversity analysis is for.
One counterintuitive finding is worth knowing: removing duplicates across crawls can reduce performance, because it preferentially retains high-entropy noise pages. Cleaning is a set of judgements, not a switch.
What should teams do instead?
Audit before training, filter aggressively, and buy verification rather than volume.
Audit a sample per language. A hundred sentences read by a speaker tells you more than the corpus size does. It is cheap, and almost nobody does it.
Treat filtering as a performance lever, not just cleanup. Model-based filtering — using a trained classifier to score and keep only higher-quality documents — has matched baseline benchmark results on as little as 15% of the tokens.
Verify language identity with speakers, not automated language identification alone, which is weakest exactly where errors cluster. This is one reason multilingual LLM training data quality programmes lean on native-speaker review rather than tooling alone.
Set a usable-token floor per language, so that volume which is not usable stops counting toward a target.
Prefer produced data where crawled data is thin. Below a point, commissioning verified in-language data is cheaper than cleaning a corpus that was never usable — the logic behind most multilingual data collection programmes and behind data validation steps that check a dataset before it reaches training.
That last point is where an operational partner matters. Lifewood's multilingual work sits on the produced-and-verified side of this line: speech and text collected and checked by screened native speakers across 50+ languages and delivered with the metadata and quality history that makes an audit possible. A dataset that arrives with evidence of what it contains is a different asset from one that arrives with only a token count.