Short answer. Three different data products get bought under the label "LLM training data", and they do different jobs. SFT (supervised fine-tuning) teaches the model what a good response looks like, and needs written demonstrations — expensive, expert-dependent, the highest-leverage per item. RLHF (reinforcement learning from human feedback) teaches the model which of two responses is better, and needs preference comparisons — cheaper per item, needs far more of them, and lives or dies on rater agreement. Distillation transfers behaviour from a stronger model into a smaller one, and needs generated data plus human validation. Most enterprise programmes need SFT first, evaluation data second, and RLHF third — and buy them in the reverse order, which is the most common and most expensive sequencing mistake.
Teams arriving at this market usually know they need "human data" and are less clear about which kind, in what proportion, and in what order. The vocabulary does not help: vendors sell all three from the same page, and the pricing units are not comparable.
This guide separates them — what each actually is, what data it consumes, how quality is measured for it, and how to sequence spend.
The three products side by side
| SFT | RLHF / preference data | Distillation | |
|---|---|---|---|
| What it teaches | What a good answer looks like | Which of two answers is better | How a stronger model behaves |
| Human produces | A written demonstration response | A ranking or comparison, with rationale | Validation and filtering of generated data |
| Cost per item | Highest | Moderate | Lowest per item, highest in compute |
| Volume needed | Lower | Higher | Highest |
| Expertise needed | High — the writer must be able to produce the target quality | Moderate to high — the rater must be able to judge it | Moderate, concentrated in review |
| Main quality risk | Inconsistent style and depth across writers | Low rater agreement; rubric ambiguity | Inheriting the teacher model's errors |
| Measured by | Rubric conformance, expert review | Chance-corrected agreement between raters | Downstream evaluation; error inheritance checks |
The economic point buried in that table: writing is harder than judging, and judging is harder than validating. Your budget should follow the difficulty, and so should your sourcing — the pools of people who can do each are different sizes and have different costs.
SFT: demonstrations
What it is. Prompt–response pairs where a human writes the response the model should have produced. It is the most direct way to move a model's behaviour, because it shows the target rather than scoring attempts at it.
Where enterprises use it. Domain adaptation — legal, medical, financial, industrial support — and voice and format conformance, where a model must produce output in a specific house style, structure or register.
What decides quality. Three things, in order:
- Writer capability. The demonstration is a ceiling. A writer who cannot produce expert-quality output produces a dataset that teaches the model to be a non-expert. For specialist domains this means qualified writers, verified rather than self-declared.
- Rubric specificity. "Be helpful and accurate" produces inconsistent data. The rubric should specify structure, depth, hedging behaviour, refusal behaviour, citation practice and formatting — anything you would notice being wrong.
- Coverage of the prompt distribution. Demonstrations should span the distribution of prompts your users actually send, including the awkward ones, not the prompts that are pleasant to answer.
How to check it before buying volume. Commission twenty items across your hardest cases from two different writers and read them side by side. Variance between writers on the same prompt is the number that predicts your dataset's consistency.
RLHF: preference data
What it is. A human sees two or more model responses and says which is better, usually with a rationale and often with per-dimension ratings — helpfulness, accuracy, safety, tone.
Where enterprises use it. Aligning behaviour where "better" is easier to recognise than to write: tone, refusal calibration, verbosity, adherence to policy. It is also how safety behaviour is tuned in practice.
What decides quality. One thing dominates, and it is measurable:
Cohen's kappa = (Observed agreement − Expected agreement) ÷ (1 − Expected agreement)
Rater agreement is the whole ballgame. If independent qualified raters disagree about which response is better, the preference signal is noise and the model learns noise. Low agreement almost always means the rubric is under-specified, not that the raters are poor — which makes it the cheapest diagnostic available and the one to run on the very first batch.
Three practical requirements:
- A per-dimension rubric, not a single "which is better". Aggregate preference collapses trade-offs — a response that is more accurate and less pleasant loses for reasons nobody recorded.
- Rationales captured. They are how you debug the rubric, and they are more informative than the preference labels themselves during the first weeks.
- Ties permitted and defined. Forcing a choice between two equivalent responses manufactures signal that is not there.
Multilingual note. Preference is culturally situated — politeness, directness and appropriate hedging differ by market. Preference data for a language should be produced by in-market native speakers, not by translating an English preference set, which produces a model that is polite in an English way in every language.
Distillation: generated data with human validation
What it is. A stronger model generates training data that trains a smaller or cheaper one. The human role moves from producing to validating and filtering — deciding what is good enough to keep.
Where enterprises use it. Cost and latency reduction, on-device or edge deployment, and expanding coverage of a task where you already have a model that performs it well.
What decides quality. The failure mode is specific and easy to miss: the student inherits the teacher's errors, including its confident ones, and human validation that is too light will pass them because they read fluently. Requirements:
- A validation rate that is stated and defended, with sampling designed to find rare errors rather than confirm common correctness.
- Error-class tracking on the teacher, so known weaknesses are screened for specifically rather than hoped against.
- Provenance labelling. Synthetic items must be identifiable in the corpus so their proportion can be controlled and their effect isolated during evaluation.
- Attention to the licence position of the teacher model's outputs for your intended use — this is a contractual question, not a technical one, and it is easier to answer before the data exists.
Evaluation data: the thing nobody budgets and everybody needs
Not a training product, and the highest-return purchase in the list. Without a held-out, human-built evaluation set, you cannot tell whether any of the above worked.
Requirements: built independently of the training data, covering the same distribution including the hard tail, refreshed periodically to limit overfitting, and — for multilingual programmes — built per language rather than translated. Contamination screening against the training corpus is worth the effort; a benchmark the model has memorised is worse than no benchmark, because it produces confident wrong decisions.
How to sequence spend
- Evaluation set first. You cannot manage what you cannot measure, and every later decision is made against this.
- SFT next, targeted narrowly at the specific behaviours that are wrong. A small, high-quality, well-covered SFT set usually beats a large diffuse one.
- Preference data third, once the rubric is stable and rater agreement has been demonstrated on a pilot batch.
- Distillation last, when behaviour is right and the goal is cheaper or faster inference.
The common inversion — buying large volumes of preference data before the rubric is stable and before an evaluation set exists — produces a dataset with low agreement, no way to prove it helped, and no diagnosis available afterwards.
What to ask a vendor
- Which of the three do you actually deliver in-house, and which do you subcontract?
- How are specialist SFT writers qualified — verified or self-declared?
- Show me rater agreement figures from a comparable preference project, per task family.
- What is your rubric development process, and who owns the rubric?
- How many rationales do you capture, and can we read them?
- For multilingual work: are preference raters in-market native speakers?
- How do you handle disagreement and edge cases — what is the escalation path?
- For distillation: what is your validation rate and how is the sample drawn?
- What is your annotator retention on projects of our length?
- What do you tell clients who ask for volume before the rubric is stable?
The last question is the most revealing. A vendor whose answer is "we would sell it to them" is selling throughput; one who describes pushing back is selling outcomes.
How Lifewood approaches this
Lifewood delivers all three in-house — RLHF, SFT and data distillation, with prompt and response evaluation across 50+ languages — through a managed workforce in owned delivery centres rather than an open crowd. That matters most for preference data: agreement figures only become meaningful when the same qualified raters stay with a rubric long enough for it to stabilise, which is precisely what high-churn sourcing prevents.
The multilingual dimension is the structural one. Preference is culturally situated, so preference data for a market should be produced in that market; 40+ delivery centres across 30+ countries and 56,788 contributors make in-market rating practical in languages where the alternative is translating an English preference set. engagements span frontier-model labs, voice-AI developers, AI compute vendors, computer-vision suppliers and autonomous-mobility programmes, with client identities withheld by agreement.
See enterprise LLM training data, type B horizontal LLM data, type C vertical LLM data, AI data validation and QA process.
Sources and further reading
- Cohen's kappa is the standard chance-corrected agreement measure for preference and judgement tasks.
- Companion guides: Multilingual LLM Training Data and What Accuracy Standard Should You Require From an Annotation Vendor?
- Lifewood LLM data scope is published at lifewood.com/enterprise-llm-training-data.