Short answer. AI training data is any material used to teach or test a model — pretraining corpora, written demonstrations, preference comparisons, annotated task data and evaluation sets — and these are different products with different producers, costs and quality tests. Quality is not one property but seven that trade against each other: correctness, consistency, coverage, relevance, freshness, diversity and provenance. The decisive question is whether the data represents real deployment conditions, not how much of it exists.
Key takeaways
- AI training data covers at least five distinct products — pretraining corpora, supervised demonstrations, preference comparisons, annotated task data and evaluation sets — each bought from different suppliers and judged by different tests.
- Data quality has seven independent dimensions: correctness, consistency, coverage, relevance, freshness, diversity and provenance; no single configuration maximises all seven at once.
- More data only helps when it adds usable signal; beyond that point it adds duplication, reinforces existing skew, and increases rights exposure without moving the aggregate metric.
- A seven-step dataset audit — from writing down the intended behaviour to running a pilot experiment on a slice of data — routinely changes what gets bought before training begins.
- Lifewood measures annotation quality against a 95%+ accuracy SLA and records provenance per item across 50+ languages and 40+ delivery centres in 30+ countries.
What counts as AI training data?
The phrase covers at least five distinct products. They are bought from different suppliers, produced by different people, and judged by different tests — which is why a single "training data" line in a budget usually hides a sequencing mistake.
Training data is any material used to teach or evaluate a model, and the five products below sit at different points in that pipeline rather than being interchangeable line items.
| Product | What it teaches | Who produces it | How quality is judged |
|---|---|---|---|
| Pre-training corpus | General language, world knowledge, format | Sourced or licensed at scale; little human authoring | Coverage against a stratification plan; deduplication; rights position |
| Supervised demonstrations | What a good response looks like | Writers capable of producing the target quality | Rubric conformance; expert review; variance between writers |
| Preference comparisons | Which of two responses is better | Trained raters, in-market for multilingual work | Chance-corrected agreement between independent raters |
| Annotated task data | Where a signal is and what it means | Trained annotators against a versioned guideline | Inter-annotator agreement; adjudication load |
| Evaluation and safety sets | Nothing — they measure | Built independently of the training set | Contamination screening; coverage of the hard tail |
Two observations that change budgets. First, evaluation data is the item most often left out and the one with the highest return, because without it no claim about the other four can be tested. Second, difficulty runs down the middle column: writing is harder than judging, and judging is harder than sourcing — so sourcing rates should never be used to reason about demonstration costs.
A dataset is also more than its records. It carries context: source, consent or licence basis, transformations applied, the annotation guideline version in force, and the checks run. That context is what makes a dataset auditable, refreshable and defensible when a model's behaviour is questioned, and it cannot be reconstructed afterwards. For a related distinction once a dataset moves from general to specialised use, see horizontal vs vertical LLM training data.
What are the dimensions of data quality?
"High quality" is not a property a dataset has. It is seven properties, each with a different test and a different failure mode.
| Dimension | The question it answers | How to test it | What it costs you when missing |
|---|---|---|---|
| Correctness | Is the label right? | Adjudicated gold set the annotators cannot identify | Direct error, learned as truth |
| Consistency | Do two annotators apply the guideline the same way? | Chance-corrected agreement on overlapped items | Unstable decision boundaries at category edges |
| Coverage | Are the conditions of deployment represented? | Stratified count against a written coverage plan | Blind spots invisible in aggregate metrics |
| Relevance | Does this material relate to the task? | Sample review against the task specification | Wasted compute; diluted signal |
| Freshness | Does it reflect current practice, products and language? | Dated sampling; review cadence per domain | Confident, fluent obsolescence |
| Diversity | Does it vary along the axes that matter? | Distribution report per axis, not overall | A model that works for the majority case only |
| Provenance | Can you say where each item came from? | Per-item origin record, exportable | Rights exposure; nothing can be filtered later |
Provenance is the recorded chain of where a data item came from, under what rights basis, and what was done to it before it reached the model.
The dimensions trade off. Narrowing a taxonomy raises consistency and loses the boundary cases. Discarding older material raises freshness and reduces volume. Sourcing more broadly raises diversity and reduces correctness until the guideline catches up. No configuration maximises all seven, which is why they should be specified per project rather than asserted as a general standard. For practical methods on two of these dimensions, see how to measure dataset diversity and how to interpret inter-annotator agreement scores.
NIST's AI Risk Management Framework (AI RMF 1.0, January 2023) makes the same argument at the programme level: trustworthiness is managed across a lifecycle rather than certified once at a benchmark. Data quality is the earliest point in that lifecycle where the decisions are cheap.
Is more training data always better?
No, and the belief that it is drives most of the waste in this market.
Additional data helps when it adds coverage the model does not have or reduces uncertainty in a region where the model is weak. It stops helping — and starts hurting — in four specific ways:
- Duplication distorts evaluation. Near-duplicates spread across a train/test split inflate measured performance without improving the model, and the effect is invisible unless you deduplicate across the split rather than within it.
- The average hides the segment. A large dataset can produce an impressive aggregate figure while performance in a rare-but-important segment is poor. The aggregate rises with volume; the segment does not.
- Skew compounds. Volume is usually acquired from whatever is easiest to source, which means each additional batch typically reinforces the existing distribution rather than filling its gaps.
- Rights exposure scales with volume. Material with unclear provenance is a liability that grows with every batch ingested, and cannot be removed later if origin was never recorded.
The better question is whether a batch increases usable signal — which in practice means collection targeted at under-represented scenarios, difficult examples, new markets and the model's own observed failures, rather than more of what it already handles.
Coverage floor is the ratio of items collected to items required in the worst-covered stratum, not the average ratio across all strata.
Coverage floor = min over strata of (items in stratum ÷ target for stratum)
Report the floor, never the mean. The mean is the number that lets a corpus with a completely empty stratum look well covered, and it is the number vendors and internal teams both default to.
How do you audit a dataset before training on it?
A dataset audit takes a few days and routinely changes what gets bought. Run it in this order, because each step makes the next cheaper.
- Write down the intended behaviour first. Task, users, languages, operating environments, the cost of each failure class, and the criteria the model will be judged on. Every later question is asked against this. A dataset cannot be evaluated in the abstract.
- Check stratification, not size. Ask for counts per stratum — language, region, device, scenario, class, difficulty — against the plan from step 1. An unstratified count is a file size.
- Sample stratified, not globally. A global pass rate is dominated by the largest and easiest segment. Draw from each stratum and report each separately.
- Hunt for leakage. Check for near-duplicates across the train, validation and test partitions, and for any item in the evaluation set that could plausibly appear in a pre-training corpus. A benchmark the model has memorised is worse than no benchmark, because it produces confident wrong decisions.
- Read a hundred items by hand. Not a report about them — the items. This is the step teams skip and the one that finds what no metric names: truncated records, boilerplate, machine-translated text presented as native, annotation that satisfies the guideline and misses its intent.
- Trace ten records end to end. Pick them at random and ask the supplier to show source, rights basis, collection method, transformations and annotation history for each. Whether the answer arrives in an hour or a fortnight tells you what the provenance record actually is.
- Run the pilot experiment. The only reliable judgement of a dataset is whether it improves the target model or evaluation under the conditions that matter. Do it on a slice before committing to the volume.
Inter-annotator agreement is a chance-corrected measure of how consistently two or more independent annotators apply the same guideline to the same item. For subjective tasks, add one more step: overlap a share of items, compute chance-corrected agreement, and interpret it against a named scale. The Landis and Koch bands (Biometrics, 1977) — 0.61–0.80 substantial, above 0.80 almost perfect — are the common reference, and were presented by their authors as arbitrary benchmarks rather than statistical thresholds. Naming the scale is part of the claim; see gold sets, audit sampling and consensus for three concrete ways to run this on annotated data.
What should a dataset record contain?
Version the documentation with the data. A workable record per release contains:
- Purpose — what the dataset was built for, and the uses it is not suitable for.
- Composition — counts by stratum, class balance, language and dialect breakdown, modality mix.
- Collection and rights — how each source was obtained, over what period, under which licence or consent basis.
- Annotation — guideline version, annotator qualification, overlap rate, agreement figures, adjudication rules. See how to write annotation guidelines that annotators actually follow.
- Known limitations — the gaps you already know about. A dataset card with no limitations section is marketing.
- Change log — what moved since the last version, and which earlier work was re-adjudicated as a result.
The point of the record is not compliance. It is that six months later, when the model behaves oddly in one market, the record is the only thing that lets anyone find out why.
How does Lifewood approach training data quality?
Lifewood treats training data as a controlled pipeline rather than a one-off delivery, with coverage specified before collection begins and quality tracked against a fixed threshold throughout rather than assessed only at handover.
Coverage is specified before collection starts, sampling is stratified rather than reported as a global pass rate, annotation guidelines are versioned, and every item passes dual-layer human review held to a 95%+ accuracy threshold. Provenance is recorded per item at intake rather than reconstructed afterwards, because reconstruction at handover is not achievable at scale.
The constraint on quality in global programmes is who is available to judge the data in-market. Lifewood works across 100+ languages through 40+ delivery centres across 30+ countries with 56,000+ registered contributors, so material is produced and reviewed in-market rather than translated into place — which matters most in exactly the strata where a coverage floor is lowest. See AI data validation and enterprise LLM training data for how this applies to specific delivery models, and multilingual LLM training data quality for how the same controls run across languages.