Skip to main content
AI Data

How Do You Know Your Evaluation Set Has Not Leaked Into Training Data?

Lifewood Team6 min read

Short answer. You cannot prove a public benchmark is clean, so you stop relying on proof and build evidence instead. The three practical methods are a held-out set written after the model was trained, a difficulty-matched twin of the benchmark you care about, and canary strings planted in your own corpus. The one thing that never works is trusting a score on a benchmark that has been on the open web for years.

Dataset contamination is when data closely resembling benchmark questions leaks into training data, so a score measures recall rather than reasoning. It is not a hypothetical. When GPT-3 was trained, its authors reported finding the training set contaminated with test data despite deliberate efforts to scrub it, and every corpus assembled from the web since has had the same exposure.

The uncomfortable part is that contamination is hard to disprove. A model provider can filter for exact matches and still miss paraphrases, translations, discussion threads quoting the questions, and tutorial blog posts working through the answers. So the question for anyone buying or building an evaluation is not "is this benchmark clean" but "what would I see if it were not".

Key takeaways

  • Contamination is not detectable by inspection of a model you do not control. It has to be inferred from behaviour.
  • The strongest evidence is a difficulty-matched twin: a new set built to mirror an existing benchmark, then a comparison of scores across the two.
  • The GSM1k study found accuracy drops of up to 8% against GSM8k, with some model families showing systematic overfitting — but also found frontier models showing minimal signs of it.
  • A positive relationship was measured between a model's probability of generating a benchmark example and its score gap, which is the closest thing to a memorisation fingerprint.
  • Canary strings are the only method that gives a clean yes or no, and they only work in a corpus you control.
  • Keeping an evaluation set private and unpublished is a control, not a courtesy.

Quick comparison

MethodWhat it answersWorks on a vendor model?Main limitation
Difficulty-matched twinDoes the score survive a fresh set?YesExpensive; needs careful matching
Randomised-answer ceilingIs performance above what capability allows?YesIndirect; needs a well-formed ceiling
Canary stringsWas this exact string in training?NoOnly works in a corpus you control
Order shufflingWas a specific serialisation memorised?Needs log-probabilitiesWeak signal; easily masked
Exact-match filteringAre there verbatim copies?Only as a providerMisses paraphrase and translation

What is dataset contamination, exactly?

Contamination is the presence in training data of material close enough to a benchmark's test items that the model can answer from memory rather than capability. It covers exact copies, paraphrases, translated versions, and third-party discussions that reproduce the items.

The distinction that matters operationally is between contamination and overfitting to a benchmark's style. A model trained on thousands of worked grade-school arithmetic problems will get better at grade-school arithmetic honestly. A model that has seen the specific test items will score better without being better. Both raise the number; only one raises the capability, and from outside the training run they look identical.

How did the GSM1k study actually test this?

Researchers commissioned a new benchmark, GSM1k, built to mirror the style and complexity of the established GSM8k grade-school maths benchmark, and then compared how models performed across the two. The two sets were matched on human solve rates, number of solution steps, answer magnitude and other metrics, so a score gap could not be explained away as a difficulty gap.

The results are more nuanced than the headline usually suggests, and the nuance is the useful part. Evaluating leading open and closed models on GSM1k showed accuracy drops of up to 8%, with several model families showing evidence of systematic overfitting across almost all model sizes. But the authors are explicit that many models, especially frontier ones, showed minimal signs of overfitting, and that all models broadly demonstrated generalisation to novel maths problems guaranteed not to be in their training data.

The memorisation evidence came from a separate analysis. The authors found a positive relationship, with a Spearman r-squared of 0.36, between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k. A model more likely to reproduce the old benchmark verbatim was more likely to drop on the new one. That is as close to a fingerprint as this field has.

The authors also kept GSM1k private rather than publishing it, specifically to limit future contamination. That decision is the method, not an afterthought.

Can you detect contamination without building a whole new benchmark?

Yes, with weaker but cheaper signals. Three approaches work without commissioning a parallel dataset, and each answers a slightly different question.

Randomised-answer benchmarks. Publish a benchmark whose answers have been randomised, so that no amount of genuine capability should beat chance-level accuracy. One proposal formalises this: even a fully capable model should not exceed the Bayes accuracy of a randomised set, so a model that surpasses that ceiling is giving a strong signal of contamination. The attraction is that it works from observed performance alone, which makes it usable against black-box models.

Canary strings. Insert unique random strings into a corpus and later test whether the model will reproduce them. This technique was proposed for auditing memorisation and is used to detect test-set contamination. It is the only method here that returns an unambiguous answer, and its limitation is severe: you can only plant a canary in data you control, so it audits your own pipeline and not a vendor's.

Order shuffling. Compare the completion log-probability of a benchmark item before and after shuffling the order of its options or elements. A model that has memorised a specific serialisation will often show a telling difference.

What should a buyer actually require?

Require that the evaluation deciding your purchase was not available for training. That single condition eliminates most of the problem, and it is a procurement question rather than a research one.

In practice that means three things. Commission or hold back a test set that postdates the models you are comparing, and do not publish it. Where you must use a public benchmark for comparability, pair it with a private twin and report both numbers, because the gap between them is more informative than either alone. And treat any vendor's self-reported score on a years-old public benchmark as a claim about familiarity with that benchmark, not as a measurement of capability.

This is where the construction of the evaluation set stops being a research concern and becomes a sourcing one. A test set that is genuinely unseen has to be newly written by people, matched for difficulty against whatever it replaces, and kept out of public circulation afterwards. Those are data-collection requirements, and they are the reason evaluation work and annotation work tend to sit with the same teams.

Frequently asked questions

They can filter, and it helps, but filtering is not a solution. Exact-match and n-gram filtering miss paraphrases, translations, partial quotations and discussions that reproduce items in different wording. The GPT-3 authors reported discovering their training set was contaminated despite efforts to scrub test data occurrences from it.

No, and this is the most common misreading. A drop can mean contamination, overfitting to a style, or simply that the new set is harder. That is why difficulty matching across human solve rates, solution steps and answer magnitudes is essential, and why the memorisation analysis was done separately from the score comparison.

Not according to the GSM1k work. It found several model families showing systematic overfitting across almost all sizes, while many models, and frontier models in particular, showed minimal signs of it. Contamination is a property of a given training pipeline rather than a property of scale.

Because publication is what creates the exposure. A benchmark on the open web will be scraped, quoted, translated and worked through in tutorials, and each of those is a contamination route. Private held-out sets exist precisely so there is something left to measure with.

Treat any public benchmark as decaying from its publication date, and expect to need a fresh private set for each major purchasing or release decision. A set that has informed a public leaderboard has already begun to lose its value as a measurement.

Sources and further reading

  1. A Careful Examination of Large Language Model Performance on Grade School Arithmetic — the GSM1k study: difficulty matching, accuracy drops of up to 8%, and the Spearman r-squared of 0.36 memorisation relationship
  2. CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting — Ishida, Lodkaew and Yamane: the randomised-answer proposal and the Bayes-accuracy ceiling argument
  3. Language Models are Few-Shot Learners — the GPT-3 paper, including its own account of discovering test-set contamination
  4. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks — the canary-string method for auditing memorisation

Have an AI or visibility project in mind?

From AI evaluation and human-in-the-loop review to GEO and AEO strategy, our team can help you deploy with confidence and get found in the AI search era.

Talk to our team