Short answer. In preference annotation the usual instinct — push inter-annotator agreement as high as it will go — destroys the signal a reward model is supposed to learn. MultiPref, 10,000 preference pairs rated by four annotators each, reported a quadratic weighted kappa of 0.268, with about 39% of pairs showing diverging preferences; PRISM found alignment preferences genuinely subjective across 1,500 participants in 75 countries. The operational problem is not raising agreement but separating drift, which is a defect, from legitimate variation, which is data — and both look identical in an agreement statistic. The fix is splitting the specification into objective components, where low agreement means drift, and subjective ones, where variation is recorded rather than resolved.
Here is the thing about running preference annotation across a global delivery network that took me a while to properly appreciate: the obvious goal is the wrong one.
In most annotation work, you drive inter-annotator agreement as high as you can. Disagreement means ambiguity, ambiguity means guideline problems, and a well-run programme converges. That instinct is correct for object detection, for transcription, for intent classification.
Apply it to RLHF preference data and you will systematically destroy the signal you were hired to collect.
Because in preference annotation, a significant share of disagreement is not error. It is people with different backgrounds, values and contexts genuinely preferring different things, which is exactly the variation a reward model should be learning about. Flatten it and you have trained a model to satisfy a consensus that no actual person holds.
Lifewood started its first LLM and RLHF programme in 2023, and this work now runs across a network of 40-plus delivery centres in 30-plus countries. The question this piece is about is the one that matters operationally: how do you keep raters calibrated across that footprint without erasing the cultural variation you are there to capture?
First, the uncomfortable baseline: agreement is low, and that is normal
If you have come to preference data from classification work, the agreement numbers will alarm you.
In MultiPref, a dataset of 10,000 preference pairs where each pair was annotated by four different annotators, researchers reported a quadratic weighted Cohen's kappa of 0.268. Roughly 39% of preference pairs showed diverging preferences after filtering out ties and slight-preference cases.
For context, on the Landis and Koch scale that would sit in the "fair" band, well below what any classification programme would accept.
And yet MultiPref is a well-constructed dataset with trained annotators and a clear protocol. The low agreement is not a quality failure. It reflects what preference judgement on open-ended responses is actually like.
The research literature has moved decisively on this point. Work going back to Aroyo and Welty, and more recently Basile and colleagues, argues that annotator disagreement is not merely measurement error: it reflects semantic ambiguity, subjective interpretation and genuine value pluralism. For RLHF specifically, how you aggregate that disagreement determines which interpretations remain visible to the learning system at all.
So the first discipline in running preference work at scale is to stop treating a low agreement number as a problem to be fixed and start asking a harder question: which part of this disagreement is signal and which part is drift?
Separating legitimate variation from actual drift
This is the core operational problem, and I think it is genuinely under-discussed in the industry.
Drift is when a rater's application of the specification changes over time or diverges from the shared standard for reasons unrelated to the content. Fatigue, boredom, a misremembered guideline, an unconscious shortcut, a local team leader's idiosyncratic interpretation that spreads through a site. This is a defect and it needs correcting.
Legitimate variation is when raters apply the specification correctly and still reach different conclusions, because the judgement being asked genuinely depends on who is answering. This is data and it needs preserving.
The two look identical in an agreement statistic. Both show up as disagreement. Telling them apart requires structure.
The mechanism that works is splitting the specification into objective and subjective components, and measuring them separately.
Objective components are things where a correct answer exists: is the response factually accurate, does it follow the stated instruction, does it contain the requested elements, does it violate a safety rule. Agreement on these should be high, and low agreement here is drift. This is where gold items work exactly as they do in classification.
Subjective components are things where the judgement is genuinely a preference: is this tone appropriate, is this level of directness helpful, is this length right, is this framing respectful. Agreement here will be lower and should be, and the variation should be recorded rather than resolved.
A well-designed preference protocol asks raters to score these separately rather than producing a single holistic "which is better" judgement. It costs more per item. It also means that when agreement drops, you can tell which kind of disagreement you are looking at.
The cultural dimension, with actual numbers
The variation across countries is real and it has been measured.
PRISM, a study by Kirk and colleagues, collected alignment preferences from 1,500 participants across 75 countries and found that preferences are subjective and context-dependent rather than universal. That is the foundational finding underneath everything in this section.
CulturalFrames, evaluating cultural expectation alignment, reported country-level Krippendorff's alpha in the range of 0.24 to 0.42 for image-prompt alignment and quality judgements, comparing that against CUBE's reported range of 0.09 to 0.58 and noting Fleiss kappa figures of 0.179 to 0.406 against CultDiff's 0.07 to 0.17. These are the agreement levels the field actually operates at on culturally-loaded judgements.
What this means practically for a distributed operation is that cross-site agreement targets have to be set per dimension, not globally. Expecting Manila, Nairobi and Belgrade to converge on tone appropriateness for a given response is expecting them to converge on a cultural judgement. Expecting them to converge on whether a response contains a factual error is entirely reasonable.
There is a subtler version of this problem worth flagging. If your rater pool is concentrated in a few countries and you aggregate to a single reward signal, you are exporting those countries' preferences to every market the model serves. That is not a neutral technical choice. It is a decision about whose norms the model encodes, made implicitly by whoever staffed the annotation programme.
The drift mechanisms that actually catch problems
With the objective and subjective split in place, here is what monitoring looks like in practice.
Shared calibration sets across all sites. A common batch of items, including gold items on the objective dimensions, is rated by every site on a recurring cycle. This is the single most important mechanism, because it produces directly comparable numbers. Without a shared batch, site-level agreement differences are confounded with batch difficulty differences and mean nothing.
Per-rater temporal tracking. A rater's agreement with the cohort is tracked over time, not just measured once. A rater whose objective-dimension accuracy declines over weeks is drifting, regardless of whether their absolute number is still above threshold. This catches fatigue and shortcut formation before they show up in delivery.
Site-level divergence monitoring. If one site's ratings on a subjective dimension diverge from the network consistently and in one direction, that is worth investigating. It may be genuine cultural signal, in which case it should be recorded as such. It may be a local guideline interpretation that took hold, which is drift with a local accent and needs correcting at team-lead level.
Repeat items with temporal separation. This one is underused and genuinely diagnostic. Insert the same item for the same rater weeks apart and measure self-consistency. A rater who disagrees with their own earlier judgement at a high rate is not producing stable preference signal.
That last mechanism connects to a finding that should make anyone in this field pause. A 2026 choice-blindness study reported that 91% of surreptitiously swapped preferences went undetected by the people who had made them.
Related work analysing PRISM and PluriHarms has examined preference inconsistency where the same annotator rates semantically similar prompts differently, arguing that some annotation responses may not represent stable underlying preferences at all.
If a rater does not notice their own choice being reversed, then the assumption that every preference label reads out a fixed internal value is shakier than most pipelines treat it. This is an argument for measuring self-consistency, weighting by preference strength, and being honest about which comparisons carry real signal.
What we actually do about the raters
Structure catches drift. Preventing it is about how the rater cohort is built and maintained.
Recruit for the market, not just the language. A rater judging tone appropriateness for Indonesian users should be Indonesian, not a fluent Indonesian speaker who has lived elsewhere for fifteen years. Cultural currency matters for exactly the dimensions where variation is legitimate.
Document rater demographics as dataset metadata. Country, language variety, age band, and any other attribute the programme has a defined analytical use for, collected with consent. Without this the client cannot later ask whether a preference pattern reflects a genuine population view or an artefact of who happened to be staffed.
Rotate calibration, not raters. High rater turnover destroys preference consistency in a way it does not destroy classification consistency, because a new rater brings a new set of priors rather than just needing to learn a rule. Retention is a quality mechanism in preference work more than anywhere else.
Run cross-site disagreement reviews, not just within-site ones. When two sites diverge on a subjective dimension, the review should include raters from both. Frequently the output is not a resolution but a documented finding that the dimension is culturally variable, which is more useful to the client than a forced consensus.
Keep the distribution, not just the aggregate. Where the client's pipeline supports it, delivering the full label distribution alongside the aggregated preference preserves information that a single scalar discards. The reward modelling literature has increasingly moved toward distributional and group-aware approaches for exactly this reason.
What good looks like
A preference programme running well across a distributed network reports, per dimension and per site:
Objective-dimension agreement, which should be high and stable. Subjective-dimension agreement, which will be lower and is monitored for change rather than level. Per-rater temporal trend on objective dimensions. Self-consistency rate on repeat items. Site-level divergence patterns, flagged and characterised as either cultural signal or drift. And rater demographic composition, so the client knows whose preferences they have bought.
That is a more complicated report than a single accuracy figure. It is also the only version that lets a client make an informed decision about whether the preference data they are training on represents the users they are building for.
The reason Lifewood runs this work through a distributed network rather than concentrating it is not primarily cost. It is that a reward model trained on preferences collected in three countries encodes three countries' norms, and a client deploying globally usually wants to know that before they find out from users.
Key takeaways
- In preference annotation, driving agreement as high as possible destroys the variation a reward model should be learning. Some disagreement is legitimate value pluralism, not error.
- MultiPref, with 10,000 preference pairs rated by four annotators each, reported a quadratic weighted Cohen's kappa of 0.268, with roughly 39% of pairs showing diverging preferences.
- Research from Aroyo and Welty onward establishes that annotator disagreement reflects semantic ambiguity, subjective interpretation and value pluralism rather than only measurement error.
- The core problem is separating drift, a defect, from legitimate variation, which is data. Both appear identically in an agreement statistic.
- The mechanism that works is splitting the specification into objective components, where low agreement means drift, and subjective components, where variation should be recorded rather than resolved.
- PRISM found alignment preferences subjective and context-dependent across 1,500 participants in 75 countries.
- CulturalFrames reported country-level Krippendorff's alpha of 0.24 to 0.42 against CUBE's 0.09 to 0.58 range.
- Cross-site agreement targets should be set per dimension. Convergence on factual accuracy is reasonable; convergence on tone appropriateness asks countries to converge on a cultural judgement.
- Drift monitoring uses shared calibration sets across sites, per-rater temporal tracking, site-level divergence monitoring and repeat items with temporal separation.
- A 2026 choice-blindness study found 91% of surreptitiously swapped preferences went undetected, which argues for measuring self-consistency and weighting by preference strength.
- Rater practices: recruit for the market not just the language, document demographics as metadata, prioritise retention, run cross-site disagreement reviews, and preserve the label distribution.
- A rater pool concentrated in a few countries and aggregated to a single reward signal exports those countries' norms to every market the model serves.
Sources and further reading
- Wang et al., "Diverging Preferences: When do Annotators Disagree and do Models Know?" (arXiv), on MultiPref, HelpSteer2, the 0.268 quadratic weighted kappa and the 39% divergence rate
- Kirk et al., PRISM, cited in "Hidden Consensus: Preference-Validity Compression in Human Feedback" (arXiv), on subjective and context-dependent preferences across 1,500 participants from 75 countries, and on Aroyo and Welty and Basile et al. on disagreement as more than noise
- "CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics" (arXiv), on country-level Krippendorff's alpha and Fleiss kappa ranges compared against CUBE and CultDiff
- "Position: RLHF May Not Reflect Genuine Preferences" (arXiv, 2026), on preference inconsistency analysis in PRISM and PluriHarms and the treatment of contested judgements in reward modelling
- Hash Block, "When RLHF Labels Quietly Drift" (Medium, March 2026), reporting the 2026 choice-blindness finding that 91% of swapped preferences went undetected, and on preference strength weighting
- "Reward-Robust RLHF in LLMs" (arXiv), on expert versus external annotator group comparison methodology in preference agreement testing
- Hershcovich et al., "Challenges and Strategies in Cross-Cultural NLP", ACL 2022, cited in the inter-annotator agreement metric literature
- Lifewood, company timeline, on the first LLM and RLHF programme in 2023 and the delivery network