How to Clean and Deduplicate a Pretraining Corpus

That is not because the pipeline is unusually aggressive. It reflects what raw web crawl actually contains: wrong language, duplicated, boilerplate, machine-generated, or too poor in quality to teach a model anything useful. Volume before curation is not a meaningful quantity on its own — the number that matters is what survives filtering.

The eight stages, in order: language identification, which discards non-target-language text and typically removes 50 to 80% of raw input; exact deduplication, which removes byte-identical documents via hashing; fuzzy deduplication, which finds near-duplicate documents using techniques such as MinHash locality-sensitive hashing rather than requiring an exact byte match; heuristic filtering on line statistics, word length distributions and punctuation ratios, using reference heuristics from the Gopher and MassiveText projects; quality classification, where a trained model scores documents and discards low scorers; PII removal, redacting emails, phone numbers, government identifiers and IP addresses in place rather than discarding the document; decontamination, removing evaluation-set content so benchmark scores still measure something real; and sharding into training-ready Parquet or JSONL.

One exception is worth knowing. The FineWeb2 team deliberately ran deduplication first, ahead of filtering, specifically so that filtering experiments could be evaluated on their own effect without deduplication influencing the results later. That is a methodology choice for running clean experiments, not a production recommendation — in production, dedup runs late because it is expensive and should not be spent on documents a cheaper filter would remove anyway.

Lee and colleagues (2022) measured byte-exact duplication rates of 6.7% in C4, 18.6% in RealNews and 21.67% in ROOTS. Those figures cover exact duplicates only; the same document reproduced with a different header or date counts as a near-duplicate, which fuzzy deduplication exists to catch.

The reason duplication matters is memorisation. Carlini and colleagues characterised a log-linear relationship between how often a sequence appears in training data and how likely a model is to reproduce it verbatim — simultaneously a privacy problem, a copyright problem and a generalisation problem. Near-duplicates are not a lesser version of that risk: Shilov and colleagues, publishing in Nature Communications in 2026, found that fuzzy duplicates contribute to memorisation at 0.8 times the rate of exact duplicates. That single finding is the argument for treating fuzzy deduplication as mandatory rather than optional.

MinHash LSH is locality-sensitive hashing computed over MinHash sketches of document shingles, a technique introduced by Broder for web-scale near-duplicate detection that remains the default for pretraining corpus deduplication. FineWeb's hyperparameters — 14 buckets of size 8, over 5-grams — have become a de facto standard; documents are clustered by signature and one representative per cluster is kept.

LSHBloom (Khan et al., 2024) replaces the MinHash-LSH index with Bloom filters and reports a twelve-times speedup at petascale. SemDeDup (Abbas et al., 2023) works on meaning rather than raw tokens, embedding documents with a lightweight sentence encoder and removing semantically redundant ones; a typical production sequence runs MinHash first to clear exact and near-lexical duplicates, then SemDeDup on what remains. SoftDedup takes a different approach again, reweighting duplicates instead of removing them, preserving the signal that content appeared frequently while reducing its dominance in training.

The compute cost is real: MinHash LSH on 100 billion tokens takes multiple weeks on a 96-core CPU cluster, which is why GPU-accelerated implementations exist for anyone curating at serious pretraining scale. Worth carrying as a caveat: all of these are approximate methods, appropriate for reducing redundancy rather than guaranteeing byte-exact matching, and results vary with implementation and hyperparameters enough that comparisons across corpora are unreliable.

The OpenGPT-X team tested both extensively and deduplicated per-dump and per-language, citing research confirming that local deduplication better preserves data diversity while still reducing redundancy. The mechanism is counterintuitive but consistent: aggressive cross-crawl deduplication preferentially retains high-entropy noise, because genuinely useful content that recurs across multiple crawls gets deduplicated away while unique, low-quality pages survive simply by being unique. The result is a corpus that is less redundant and worse.

Different projects land in different places. FineWeb runs per-snapshot MinHash across 96 Common Crawl snapshots to produce 15 trillion tokens — a purely local approach. FineWeb2 deduplicates globally within each language, a middle position: global within a language, partitioned across languages. Zyda takes the opposite extreme, running cross-dataset high-aggression deduplication with LSH on 13-grams. This is a live design decision rather than a settled one, though the direction of recent evidence favours partitioned approaches over fully global ones.

Run English-style shingling over Thai, Chinese or Japanese text and the resulting MinHash signatures are meaningless: real duplicates go undetected and unrelated documents get falsely clustered together. The pipeline still runs, still reports a deduplication rate, and that number is fiction. The FineWeb2 team addressed this directly by using word-level tokenizers per language to generate word n-grams, rather than one tokenisation approach for a whole multilingual corpus.

The same failure mode propagates through the rest of the pipeline. Heuristic filters built on English assumptions — average word length, punctuation ratios, stopword frequency — misfire on languages with different morphology and discard valid text as low quality. Language identification performs worse on low-resource and code-switched text, routing documents to the wrong partition or dropping them outright. Quality classifiers trained on English educational content do not transfer either. The practical result is a corpus that is well curated in English and unpredictably curated everywhere else, invisible in any aggregate statistic. The remedy is per-language configuration and per-language reporting of tokenisation, filter thresholds, language ID confidence, and dedup and pass rates — more work, and the only way to know what a multilingual training corpus actually contains once curation finishes. Building horizontal training data across many languages at once makes this failure mode more likely, not less, unless it is checked for explicitly.

Contamination usually arrives through duplication of benchmark content across the open web rather than anyone deliberately including a test set — a popular benchmark question quoted in a hundred blog posts is in the corpus a hundred times over, and exact matching against the original benchmark file will not catch the paraphrased copies. Decontamination also has to run against every benchmark a team intends to report on, which means that list has to be fixed before curation starts, not chosen after training finishes.

Web crawl over-represents what the web itself produces: written, formal, English-dominant text from a handful of high-resource languages. An audit of 205 web-crawled language corpora found at least 15 with no usable text at all and 87 falling below 50% usable content — no amount of MinHash tuning fixes a corpus that was never usable in the first place. A pass rate of 6% in English and 0.4% in a low-resource language is not one pipeline working consistently; it is a signal that those languages need collection, not further filtering. Below a certain usability threshold, commissioning produced and verified training data is cheaper than repeatedly cleaning crawled data that was never going to work, and it is where the discipline connects to broader questions of measuring dataset diversity and avoiding the kind of degradation covered in research on training models on synthetic or low-quality data.

Lifewood works in AI data and multilingual data collection, including managed multilingual data collection and enterprise LLM training data programmes, and while corpus-scale deduplication is an engineering discipline rather than an annotation one, the two meet at exactly this point: cleaning a crawl and commissioning data to fill its gaps are complementary steps, not substitutes for each other. The practical framing that follows from this: run the pipeline first, then audit per language what actually survived, rather than trusting a single aggregate pass-rate number to describe a multilingual corpus.

Report the pass rate per stage and per language, not just in aggregate, since an aggregate figure hides exactly the languages and domains that need attention. Run fuzzy deduplication in addition to exact deduplication, since fuzzy duplicates drive memorisation at roughly 0.8 times the rate of exact ones and exact hashing misses them entirely. Default to local deduplication, per crawl dump and per language, unless testing on the specific corpus in hand shows global deduplication performs better. Use per-language tokenisation for shingling rather than one whitespace tokeniser applied everywhere, since running English tokenisation over non-Latin scripts produces a deduplication rate that is effectively fiction. Order the eight production stages cheap to expensive to save compute, but consider running deduplication first instead when the goal is running clean filtering experiments rather than production curation. Fix the benchmark list before curation begins, so decontamination can run against everything a team intends to report on later, rather than being decided retroactively. Budget for GPU acceleration once a corpus is above roughly 100 billion tokens, since MinHash LSH at that scale takes multiple weeks on a CPU-only cluster. And audit what survived the pipeline, not only what was removed by it, because the corpus that remains after cleaning is bounded entirely by the crawl a team started with, and gaps in coverage are invisible in cleaning statistics alone.