Short answer. A full FineWeb-Edu-style pipeline on raw Common Crawl passes 5 to 7% of what goes in — 100 trillion raw tokens yields 5 to 7 trillion training tokens, and language identification alone discards 50 to 80% before any quality filter runs. The eight production stages are language ID, exact dedup, fuzzy dedup, heuristic filtering, quality classification, PII removal, decontamination and formatting, and the order is load-bearing. Duplication is worth measuring rather than assuming: Lee et al. (2022) found byte-exact duplication at 6.7% in C4, 18.6% in RealNews and 21.67% in ROOTS.
Pretraining Corpus?
Start with the number that frames everything else.
A full FineWeb-Edu-style curation pipeline applied to raw Common Crawl has a combined pass rate of 5 to 7%.
Start with 100 trillion raw tokens and you finish with 5 to 7 trillion training tokens.
Ninety-three to ninety-five percent of the input is discarded. Not because the pipeline is aggressive, but because that is genuinely what raw web crawl looks like: wrong language, duplicated, boilerplate, machine-generated, or simply too poor to learn from.
That figure should reframe how anyone thinks about "we have a lot of data." Volume before curation is not a meaningful quantity. The number that matters is what survives.
The eight stages, and why the order is not arbitrary
The production sequence is well established and the ordering principle is simple: run cheap operations first to minimise the data processed by expensive ones.
Language identification. Discards non-target language, typically 50 to 80% of the input. This goes first because it is cheap and removes the most volume.
Exact deduplication. Byte-identical documents, removed by hashing.
Fuzzy deduplication. Near-duplicates via MinHash LSH.
Heuristic filtering. Line statistics, word length distributions, punctuation ratios. The Gopher and MassiveText heuristics remain the reference set.
Quality classification. A model scores documents and low scorers are discarded.
PII removal. Emails, phone numbers, government identifiers and IP addresses redacted in place rather than discarded.
Decontamination. Removing evaluation set content so benchmarks still measure something.
Sharding into training-ready Parquet or JSONL.
Each stage reduces the corpus, and putting an expensive stage before a cheap one wastes compute on documents that were always going to be discarded.
One important caveat on that ordering, from FineWeb2. Because deduplication is computationally expensive, it is typically applied last. The FineWeb2 team deliberately ran it first, so that when they ran filtering experiments they could observe final dataset performance directly without deduplication later influencing the results.
That is a methodology decision rather than a production one, and it is worth knowing the distinction. If you are running experiments to decide on filters, dedup first so your results are clean. If you are running production at scale, dedup late so you are not deduplicating documents you will discard anyway.
How much duplication is actually in there
The foundational measurement comes from Lee and colleagues in 2022, and it is worth knowing because it calibrates expectations.
Byte-exact duplication rates by corpus: C4 at 6.7%, RealNews at 18.6%, ROOTS at 21.67%.
Those are exact duplicates only. Near-duplicates, which the same document reproduced with a different header or a changed date, are substantially more common and are what fuzzy deduplication exists to catch.
The reason it matters is memorisation. Carlini and colleagues characterised log-linear memorisation scaling with duplication: the more times a sequence appears in training data, the more likely the model is to reproduce it verbatim.
That is a privacy problem, a copyright problem and a generalisation problem simultaneously.
And near-duplicates are not a lesser version of the problem. Shilov and colleagues, publishing in Nature Communications in 2026, demonstrated that fuzzy duplicates contribute to memorisation at 0.8 times the rate of exact duplicates. Eighty percent of the effect, from documents that exact matching will not catch. That single finding is the argument for fuzzy deduplication being mandatory rather than optional.
The methods, and what each actually does
MinHash LSH is the workhorse. Locality-sensitive hashing over MinHash sketches, introduced by Broder for web-scale nearduplicate detection, remains the dominant production approach for pretraining corpus deduplication.
The FineWeb hyperparameters have become something of a de facto standard and are worth recording: 14 buckets of size 8, over 5-grams. Documents are clustered by signature and one representative per cluster is kept.
LSHBloom (Khan et al., 2024) replaced the MinHash-LSH index with Bloom filters, reporting a twelve-times speedup at petascale.
SemDeDup (Abbas et al., 2023) works on meaning rather than tokens, identifying and removing semantically redundant documents via embeddings. A typical production sequence runs MinHash first to clear exact and near-lexical duplicates, then embeds remaining documents with a lightweight sentence encoder such as all-MiniLM-L6-v2, chunking documents that exceed the encoder's input length.
SoftDedup takes a different approach entirely, reweighting duplicates rather than removing them, which preserves the signal that something appeared frequently while reducing its dominance.
GPU-accelerated implementations exist because the compute is otherwise prohibitive. MinHash LSH on 100 billion tokens takes multiple weeks on a 96-core CPU cluster, which makes CPU-only curation impractical at serious pretraining scale.
One caveat worth carrying: all of these approximate methods are approximate. That is fine for pretraining, where the goal is reducing redundancy rather than guaranteeing byte-stability, but it means results vary with implementation and hyperparameters in ways that make cross-corpus comparisons unreliable.
Local versus global: the debate that matters most
This is the decision with the largest effect on your final corpus, and the consensus has shifted.
Global deduplication removes duplicates across the entire corpus: across all crawl snapshots, across all languages, across all sources.
Local deduplication operates within a partition, typically per crawl dump and per language.
The intuition says global is better, because it removes more duplication. The evidence says otherwise.
The OpenGPT-X team, after extensive testing, deduplicated per-dump and per-language, and cited newer research confirming that local deduplication is more favourable than global approaches for preserving data diversity while minimising redundancy.
The mechanism, which I have written about before in this series, is counterintuitive but consistent. Aggressive cross-crawl deduplication preferentially retains high-entropy pages, because genuinely useful content that appears in multiple crawls gets deduplicated away while noisy, unique, low-quality pages survive by virtue of being unique. You end up with a corpus that is less redundant and worse.
FineWeb runs per-snapshot MinHash across 96 Common Crawl snapshots to produce 15 trillion tokens. FineWeb2 deduplicated globally per language, which is a middle position: global within a language, partitioned across languages.
Different projects land differently, and Zyda takes the opposite approach with cross-dataset high-aggression deduplication using LSH on 13-grams. The honest summary is that this is a live design decision rather than a settled one, and the direction of recent evidence favours partitioned approaches.
The multilingual problem, which is where most pipelines quietly break
Here is the detail that gets skipped in almost every pipeline description, and it invalidates results when missed.
MinHash operates on n-gram shingles, and n-grams require tokenisation. Whitespace tokenisation works acceptably for English and badly or not at all for languages that do not delimit words with spaces, use rich morphology, or write in scripts where word boundaries are not marked.
The FineWeb2 team addressed this explicitly: they used word-level tokenizers per language to obtain word n-grams, rather than applying one tokenisation approach across the whole multilingual corpus.
Run English shingling over Thai, Chinese or Japanese text and your MinHash signatures are meaningless. Duplicates are not detected and non-duplicates are falsely clustered. The pipeline runs, produces output, reports a deduplication rate, and the number is fiction.
The same problem propagates through the rest of the pipeline. Heuristic filters built on English assumptions, average word length, punctuation ratios, stopword frequency, misfire on languages with different morphological structure, discarding valid text as low quality. Language identification performs worse on low-resource languages and on code-switched text, so documents get routed to the wrong language partition or dropped entirely. Quality classifiers trained on English educational content do not transfer.
The practical consequence is that a multilingual corpus processed through an English-designed pipeline is not equally curated across languages. It is well curated in English and unpredictably curated everywhere else, and nothing in the aggregate statistics reveals this.
The remedy is per-language configuration and per-language reporting: tokenisation, filter thresholds, language ID confidence, deduplication rate and pass rate, all reported by language rather than in aggregate. That is more work. It is also the only way to know what you actually have.
Decontamination, which is easy to underestimate
One stage deserves separate attention because getting it wrong invalidates everything downstream.
Decontamination removes evaluation set content from the training corpus. If benchmark questions appear in pretraining data, benchmark scores measure memorisation rather than capability, which is the problem I have written about in the context of multilingual benchmarks becoming unreliable as models are trained to pass them.
Two practical points.
Contamination arrives through duplication of benchmark content across the web, not through anyone deliberately including a test set. A popular benchmark question quoted in a hundred blog posts is in your corpus a hundred times over, and exact matching against the original benchmark file will not catch the paraphrases.
And decontamination has to run against every benchmark you intend to report on, which means the list has to be fixed before curation rather than chosen after training.
Where our own work touches this
Declaring the interest: Lifewood works in AI data and multilingual data collection, and while corpus-scale deduplication is an engineering discipline rather than an annotation one, the two meet at a specific point that is worth naming.
Cleaning tells you what to remove. It cannot tell you what is missing.
A pipeline can be perfectly configured and still produce a corpus that is thin in exactly the languages, domains and registers you needed, because filtering only operates on what the crawl found. Web crawl over-represents what the web produces, which is written, formal, English-dominant text from a handful of high-resource languages.
We have written elsewhere in this series about the audit of 205 web-crawled language corpora that found at least 15 with no usable text at all and 87 falling below 50% usable content. No amount of MinHash tuning fixes a corpus that was never usable. Below a certain threshold, commissioning produced and verified data is cheaper than cleaning crawled data that was never going to work.
The practical framing we use with clients: run the pipeline, then audit per language what survived. A pass rate of 6% in English and 0.4% in Sinhala is not one pipeline working consistently. It is a signal that the low-resource languages in your corpus need collection, not filtering.
What to do Report the pass rate per stage and per language. Aggregate pass rates hide everything that matters.
Do fuzzy deduplication, not just exact. Fuzzy duplicates drive memorisation at 0.8 times the rate of exact ones, and exact matching misses them entirely.
Default to local deduplication, per dump and per language, unless you have measured that global performs better on your corpus.
Use per-language tokenisation for shingling. English whitespace tokenisation over non-Latin scripts produces meaningless signatures and a deduplication rate that is fiction.
Order stages cheap to expensive in production, but consider dedup-first when running filter experiments so results are not confounded.
Fix your benchmark list before curation so decontamination can run against all of it.
Budget for GPU acceleration if you are above roughly 100 billion tokens, since MinHash LSH at that scale takes weeks on CPU.
Audit what survived, not just what was removed. The corpus you have after cleaning is defined by the crawl you started with, and the gaps are invisible in the cleaning statistics.
Key takeaways
- A full FineWeb-Edu-style pipeline on raw Common Crawl has a combined pass rate of 5 to 7%. 100 trillion raw tokens yields 5 to 7 trillion training tokens.
- The eight production stages are language ID, exact dedup, fuzzy dedup, heuristic filtering, quality classification, PII removal, decontamination and sharding. Cheap stages run first.
- Language identification alone typically discards 50 to 80% of raw input.
- Lee et al. (2022) measured byte-exact duplication at 6.7% in C4, 18.6% in RealNews and 21.67% in ROOTS.
- Carlini et al. characterised log-linear memorisation scaling with duplication, making dedup a privacy, copyright and generalisation issue simultaneously.
- Shilov et al. (Nature Communications, 2026) found fuzzy duplicates contribute to memorisation at 0.8 times the rate of exact duplicates, which makes fuzzy dedup mandatory rather than optional.
- MinHash LSH remains the dominant production method. FineWeb's de facto standard hyperparameters are 14 buckets of size 8 over 5-grams.
- LSHBloom reported a twelve-times speedup at petascale by replacing the LSH index with Bloom filters. SemDeDup works on embeddings, SoftDedup reweights rather than removes.
- MinHash LSH on 100 billion tokens takes multiple weeks on a 96-core CPU cluster, making CPU-only curation impractical at scale.
- OpenGPT-X found after extensive testing that local deduplication, per dump and per language, preserves data diversity better than global approaches.
- Aggressive cross-crawl deduplication preferentially retains high-entropy noise, because useful content appearing in multiple crawls is removed while unique low-quality pages survive.
- FineWeb2 deduplicated globally per language and used per-language word-level tokenizers to generate n-grams.
- English whitespace tokenisation applied to languages without space-delimited words produces meaningless MinHash signatures and a deduplication rate that is fiction.
- Heuristic filters, language ID and quality classifiers built on English assumptions all degrade on other languages, so a multilingual corpus processed through an English pipeline is unevenly curated in ways aggregate statistics conceal.
- FineWeb2 ran deduplication first rather than last so filtering experiments could be evaluated without dedup confounding the results. In production the opposite order is usually correct.
- Decontamination must run against every benchmark you intend to report, and the list must be fixed before curation.
- Cleaning tells you what to remove but never what is missing. Below a quality threshold, commissioning verified data is cheaper than cleaning crawled data that was never usable.
Sources and further reading
- Spheron, "AI Pretraining Data Curation on GPU Cloud: NeMo Curator, Datatrove and FineWeb-Style Pipelines (2026 Guide)", on the eight-stage pipeline, the 5 to 7% combined pass rate, the 50 to 80% language ID drop rate and CPU compute requirements for MinHash LSH
- "Byte-Exact Deduplication in Retrieval-Augmented Generation", arXiv, on Lee et al. (2022) duplication rates for C4, RealNews and ROOTS, Carlini et al. on memorisation scaling, and Shilov et al. (Nature Communications 2026) on fuzzy duplicate memorisation rates
- "Merlin: Deterministic Byte-Exact Deduplication", arXiv, on MinHash LSH as the dominant production approach, LSHBloom's twelve-times petascale speedup, and SemDeDup, D4 and SoftDedup
- "FineWeb2: One Pipeline to Scale Them All", arXiv, on per-language word-level tokenizers, global per-language deduplication, the FineWeb hyperparameters and running deduplication first for experimental cleanliness
- "Data Processing for the OpenGPT-X Model Family", arXiv, on local versus global deduplication testing and the finding that local preserves data diversity better
- "ManufactuBERT: Efficient Continual Pretraining for Manufacturing", arXiv, on the practical MinHash-thenSemDeDup sequence and encoder choice
- Emergent Mind, "FineWeb-Edu-Dedup", on multi-stage deduplication, Jaccard thresholds and corpus scale, and "RefinedWeb Dataset" on comparative pipeline approaches including Zyda
- Kreutzer et al., "Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets", TACL, on corpora with no usable text, discussed earlier in this series
- Lifewood, multilingual data collection and AI data services