Short answer. Yes, and in multilingual work it frequently does. A manual audit of 205 web-crawled language corpora found at least 15 containing no usable text at all and 87 falling below 50% usable content. Adding data of that standard teaches a model the wrong language, the wrong facts or nothing at all, while consuming the training budget that verified data would have used. Below a quality threshold, volume stops helping and starts costing.
How can adding data make a model worse?
Through three mechanisms, all of which get stronger as a language gets smaller.
It crowds out good data. Training budgets are finite, so every token spent on a garbled document is one not spent on a correct one, and in a low-resource language correct documents are the scarce asset.
It teaches the wrong thing. A corpus labelled as one language but containing another does not simply fail to help. It trains a false association.
It amplifies whatever dominates. Duplicated content is learned disproportionately, so boilerplate and scraped repetitions become the model's idea of the language.
The asymmetry matters. In English, noise is diluted by enormous volumes of clean text. In a language with a few hundred thousand usable sentences, a noisy corpus can be most of what the model sees.
How bad is web-crawled multilingual data really?
Worse than most teams assume, and the evidence has been public since 2022.
Kreutzer and colleagues manually audited 205 language-specific corpora released with five major public datasets:
CCAligned, ParaCrawl, WikiMatrix, OSCAR and mC4. The findings are worth reading carefully, because these datasets underpin a great deal of multilingual model training.
At least 15 corpora contained no usable text at all, meaning not a single correct sentence in the audited sample. 87 languages fell below 50% usable data. In CCAligned, 44 of the 65 audited languages had under 50% correct sentences, and in WikiMatrix 19 of 20 did. For WikiMatrix, roughly two-thirds of audited samples were misaligned on average, with sentence pairs that looked structurally similar while describing different facts.
Separately, 82 corpora were mislabelled or used nonstandard or ambiguous language codes, meaning the dataset was not reliably about the language it claimed.
The authors added a point that should be encouraging: these problems are easy to detect, even for people who do not speak the language fluently. Nobody had checked.
What does the noise actually consist of?
Four recurring categories, each requiring a different fix.
Duplication. Redundancy in crawled data is extensive: work on Indic corpora reports it across roughly 70% of crawled pages, and one Portuguese pipeline removed around 40% of remaining pages by deduplicating within a single crawl.
Wrong or mislabelled language. Content in a different language, in a romanised variant of the claimed one, or under an ambiguous code. This is the failure that makes a dataset actively misleading.
Misalignment. In parallel corpora, sentence pairs that are not translations of each other. A model trained on these learns false equivalences with confidence.
Non-language content. Boilerplate, navigation, autogenerated text, code fragments and spam, all present at material rates in the audited low-resource corpora.
One counterintuitive finding is worth knowing: removing duplicates across crawls can reduce performance, because it preferentially retains high-entropy noise pages. Cleaning is a set of judgements, not a switch.
What should teams do instead?
Audit before you train, filter aggressively, and buy verification rather than volume.
Audit a sample per language. A hundred sentences read by a speaker tells you more than the corpus size does. It is cheap, and almost nobody does it.
Treat filtering as a performance lever. Model-based filtering has matched baseline benchmark results on as little as 15% of the tokens.
Verify language identity with speakers, not automated language identification alone, which is weakest exactly where errors cluster.
Set a usable-token floor per language. Volume that is not usable should not count toward a target.
Prefer produced data where crawled data is thin. Below a point, commissioning verified in-language data is cheaper than cleaning a corpus that was never usable.
That last point is where an operational partner matters. Lifewood's multilingual work exists on the produced-and-verified side of this line: speech and text collected and checked by screened native speakers across 50+ languages and delivered with the metadata and quality history that makes an audit possible. A dataset that arrives with evidence of what it contains is a different asset from one that arrives with a token count.
Key takeaways
- More data hurts when it crowds out good data, teaches false associations or amplifies duplicated content, and the effect is strongest in low-resource languages.
- An audit of 205 web-crawled corpora found at least 15 with no usable text, 87 below 50% usable, and 82 mislabelled or using ambiguous language codes.
- 44 of 65 audited CCAligned languages and 19 of 20 WikiMatrix languages fell under 50% correct sentences.
- Noise takes four forms: duplication, wrong language, misalignment in parallel data, and non-language content.
- Deduplication across crawls can reduce performance by retaining high-entropy noise, so cleaning is a judgement rather than a switch.
- Filtered multilingual corpora have matched baseline results on as little as 15% of the tokens.
- Audit a sample per language with a speaker, verify language identity, set a usable-token floor and prefer produced data where crawled data is thin.
Sources and further reading
- Kreutzer et al., "Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets", TACL
- "Enhancing Multilingual LLM Pretraining with Model-Based Data Selection", arXiv
- "Building High-Quality Datasets for Portuguese LLMs", arXiv
- "Pretraining Data and Tokenizer for Indic LLM", arXiv
- Lifewood, multilingual data collection