Short answer. Preference data is only as good as the agreement between the people producing it, and low agreement is almost always a rubric problem rather than a rater problem. A rubric that produces agreement specifies four things a single "which is better?" question cannot: the dimensions being judged, anchors describing what each score level looks like, a precedence rule for when the dimensions disagree, and an explicit tie policy. Add calibration rounds against adjudicated gold comparisons, capture rationales, and measure chance-corrected agreement per dimension rather than overall. Diagnose all of it on the first pilot batch — a rubric that has not been tested for agreement before volume production is a rubric that will produce noise at volume, and the noise is indistinguishable from signal once it is in the file.
Reinforcement learning from human feedback turns human judgement into a training signal. The pipeline is well documented — supervised fine-tuning on demonstrations, then human comparisons between model outputs, then a reward signal or preference-learning objective derived from them, as set out in Ouyang et al.'s work on instruction-following models. What is much less documented is the part that decides whether the resulting data is worth anything: the instrument the humans are given.
This guide is about that instrument. Which of the three LLM data products to buy, and in what order, is a separate question covered in what to buy: RLHF, SFT or distillation.
Why do raters disagree about model outputs?
Language generation is open-ended, so two responses can both be acceptable for different reasons and one response can be stronger on one dimension while weaker on another. But "the task is subjective" is a description, not a diagnosis. Disagreement has causes, and they have different fixes.
| Cause | What it looks like | Fix |
|---|---|---|
| Genuine equivalence | Raters split evenly, rationales agree that both are fine | Permit and define ties |
| Dimension conflict | A is more accurate, B is more helpful; raters weight differently | Score per dimension; state precedence |
| Missing context | Raters imagine different users or situations | Supply the context in the item, not in the rubric |
| Expertise gap | Rater cannot verify the domain claim being made | Route to a specialist tier |
| Rubric ambiguity | Rationales cite different criteria for the same choice | Rewrite the rubric with anchors |
| Rater drift | Agreement falls over weeks with no rubric change | Re-calibrate; check fatigue and cohort changes |
Only the first is inherent to the task. The other five are addressable, and four of them are addressable before production starts.
The single most useful thing a preference programme can capture is the rationale, not the preference. During the first weeks the rationales are more informative than the labels: they are how you tell dimension conflict from rubric ambiguity, and they are the material from which the next rubric version is written.
What a preference rubric has to specify
Dimensions, not a verdict. Ask which response is better on accuracy, on instruction following, on helpfulness, on safety, on tone, on completeness — separately. An aggregate preference collapses trade-offs: a response that is more accurate and less pleasant loses for reasons nobody recorded, and the model learns the aggregate.
Dimension conflict rate = Comparisons where dimensions disagree ÷ Total comparisons
Compute it on the pilot. A high conflict rate means an aggregate preference label is not meaningful for your task at all, and the programme should be collecting per-dimension scores. A low one means aggregation is safe and cheaper.
Anchors for each level. "Rate helpfulness 1–5" produces a rater's private scale. An anchor describes what a 2 looks like and what a 4 looks like, with a worked example of each. Anchors are the difference between a rubric and a questionnaire, and writing them is where most of the effort in rubric design goes.
A precedence rule. When dimensions disagree, which wins? Accuracy over tone is a common default; safety over everything is a common absolute. Whatever the answer, it must be in the document, because otherwise every rater invents one — and their private rules will be consistent within each rater and different between them, which is exactly the pattern that produces low agreement with high individual reliability.
A tie policy. Forcing a choice between two equivalent responses manufactures signal that is not there and injects it into training. Permit ties, define what qualifies as one, and expect a meaningful proportion of them on a well-constructed comparison set.
Explicit handling for refusal, hedging and length. These three are where rubrics are most often silent and where raters most reliably differ. Should a correct refusal beat a helpful answer to a borderline request? Is a hedged answer better than a confident wrong one? Does length count as thoroughness or as padding? Unstated, these become the model's behaviour anyway — decided by whichever way the raters happened to lean.
The context the rater needs, supplied in the item. Who is asking, what for, in what setting. If the rater has to imagine it, different raters imagine differently and the disagreement that follows is not about the responses at all.
How to run calibration
Calibration is not training. It is the process of finding out where the rubric is ambiguous, using raters as the instrument, before their output is treated as data.
- Build an adjudicated gold set. Twenty to fifty comparisons, including the hardest and most ambiguous, resolved by senior reviewers with written reasoning. This is the reference the rubric is calibrated against.
- Run a blind round. Every rater independently judges the gold set against the current rubric version, with rationales.
- Compute agreement per dimension, chance-corrected, and against the gold answers separately from against each other.
Cohen's kappa = (Observed agreement − Expected agreement) ÷ (1 − Expected agreement)
- Read the divergent rationales together. This is the actual work of the round. Where raters cite different criteria for the same choice, the rubric is ambiguous. Where they cite the same criterion and reach different answers, the anchors are.
- Revise the rubric, version it, and repeat until agreement stabilises on the dimensions that matter. Two or three rounds is normal; needing more than that usually means the dimension set is wrong rather than the wording.
- Re-calibrate on a cadence, and always after a rubric change, a cohort change or an observed agreement decline.
Interpret kappa against a named scale rather than a private one. The Landis and Koch bands (Biometrics, 1977) — 0.61–0.80 substantial, above 0.80 almost perfect — are the common reference, and were offered by their authors as arbitrary benchmarks rather than statistical thresholds. Objective dimensions such as instruction following should reach the higher band; genuinely subjective dimensions such as tone rarely will, and a suspiciously high figure on a subjective dimension usually indicates anchoring on a suggested answer rather than excellent work.
Triaging disagreement during production
Once volume starts, disagreement needs a routing rule rather than a discussion. Ask three questions in order:
- Is the item bad? Two responses that are near-identical, or a prompt that is ambiguous, produce disagreement that says nothing about the raters or the rubric. Remove the item and check how it entered the set.
- Is the rubric silent? If the rationales cite a consideration the rubric does not mention, that is a ruling to be written — and, once written, applied back to the affected items already collected.
- Is the rater diverging? A single rater consistently out of line with the cohort is usually a correctable misunderstanding rather than a performance problem. It is visible only if agreement is tracked per rater as well as per dimension.
Track agreement over time. A falling trend across the cohort usually means fatigue or drift; a step change usually means a rubric or item-source change that was not flagged.
Multilingual note. Preference is culturally situated — expectations about directness, politeness, hedging and appropriate detail differ by market. A preference set for a language should be produced by in-market native speakers rather than translated from an English set, which trains a model to be polite in an English way in every language. This is not a nicety: it shows up as consistently low agreement in some markets when a translated rubric is used, and the low agreement is caused by the rubric rather than by the raters.
Keep training and evaluation preferences separate
The same judgement infrastructure supports both, and the datasets must not. Training preferences shape behaviour; evaluation preferences estimate whether the behaviour improved, on held-out prompts including the difficult and adversarial ones.
Three requirements: build the evaluation set independently rather than by splitting the training set, screen it for contamination against the training corpus, and refresh it periodically to limit overfitting. For multilingual programmes, build it per language rather than translating it — an evaluation set that was translated measures the translation as much as the model.
A mature workflow closes the loop with an error taxonomy: evaluation failures are classified, and the classes with the most failures determine what the next round of preference collection targets. Without that, preference data accumulates without ever becoming more targeted.
How Lifewood approaches this
Lifewood runs preference and response-evaluation work rubric-first: per-dimension scoring with written anchors, an explicit precedence rule and tie policy, adjudicated gold comparisons, calibration rounds before volume, and chance-corrected agreement measured per dimension and tracked per rater. Rationales are captured as standard, because they are what makes the rubric improvable.
The delivery model is what makes agreement figures meaningful over time. A managed workforce in owned delivery centres rather than an open crowd means the same qualified raters stay with a rubric long enough for it to stabilise, which high-churn sourcing prevents by construction — and 414,120 training hours in 2025 is what sustains that consistency as programmes scale. For multilingual preference work, 50+ languages, 40+ delivery centres across 30+ countries and 56,788 registered contributors make in-market rating practical where the alternative is translating an English preference set. Scope covers RLHF, SFT, data distillation and prompt and response evaluation. The AI-data heritage runs to 2004, with the current company established in 2018.
See enterprise LLM training data, AI data validation, the QA process and type C vertical LLM data.
Sources and further reading
- Ouyang et al., "Training language models to follow instructions with human feedback", arXiv:2203.02155, 2022 — the demonstration-and-ranking pipeline described above.
- Landis, J.R. and Koch, G.G., "The Measurement of Observer Agreement for Categorical Data", Biometrics 33(1), 159–174, 1977 — the origin of the agreement bands quoted.
- Companion guide: What to Buy: RLHF, SFT or Distillation.