Short answer. Inter-annotator agreement measures how consistently different people label the same data. The two most widely used metrics are Cohen's kappa, which corrects raw agreement for the share that would happen by chance between two annotators, and Krippendorff's alpha, which generalises across any number of annotators, handles missing data and works across different measurement types. Both produce values from 0 to 1, but the thresholds that matter depend entirely on the task, and both measure reliability, not correctness. A high score means annotators agree; it does not mean they are right.
Why isn't raw percentage agreement enough?
Because chance inflates it, and chance inflation is the exact problem you are trying to measure past.
Imagine two annotators labelling a binary classification task where 90% of items belong to Class A. Both annotators can score 81% raw agreement simply by guessing Class A every time, without ever engaging with the actual data. The raw percentage looks reasonable. The agreement is meaningless.
This is the foundational problem Cohen's kappa was designed to address. Jacob Cohen introduced the kappa statistic in 1960 specifically to measure agreement between two psychiatric diagnosticians rating patient symptoms, correcting for the base rate of chance agreement that inflated raw percent agreement scores.
The same inflation problem appears in multilingual annotation. A dataset where one label category dominates, which is common in sentiment, toxicity and intent classification tasks, will produce high raw agreement even when annotators are applying the labels inconsistently. The chance-corrected metrics are the ones that expose this.
What does Cohen's kappa actually calculate?
The ratio of observed agreement above chance to the maximum possible agreement above chance. It answers: of the agreement that could not have happened by chance, how much actually did?
The formula is straightforward. Kappa equals observed agreement minus expected agreement, divided by one minus expected agreement. Expected agreement is calculated from the marginal distributions of each annotator's labels: if Annotator A labels 60% of items positive and Annotator B labels 55% positive, the expected chance agreement is the probability that two independent raters would land on the same label under those distributions.
Cohen's kappa yields a value from minus 1 (perfect disagreement) to 1 (perfect agreement), with 0 indicating chance-level agreement.
Key limitations worth knowing:
Cohen's kappa is for exactly two annotators. Fleiss's kappa extends this to multiple annotators using the same formula structure, but it requires every annotator to label every item. Krippendorff's alpha is used when Fleiss's kappa is not applicable, for example for variable annotator subsets where not every item is annotated by the same people.
Kappa is sensitive to label distribution. This is known as the kappa paradox: when one label dominates the distribution, kappa can be low even when annotators are quite consistent, because the expected agreement is very high and there is little room for the observed agreement to exceed it. A kappa of 0.40 on a heavily skewed task can represent the same underlying consistency as a kappa of 0.70 on a balanced one.
It measures nominal categories only in its standard form. Ordinal scales, continuous ratings and spans require different treatment.
What is Krippendorff's alpha and when should you use it?
Alpha is the more general metric. It handles any number of annotators, tolerates missing data, and works across nominal, ordinal, interval and ratio scales. In large annotation pipelines it is almost always the right choice.
Klaus Krippendorff developed alpha in 1970 for content analysis in communication research, where multiple coders categorised media content. His 2004 formalisation established alpha as the most general-purpose reliability metric, handling arbitrary numbers of coders, missing data, and multiple measurement levels, all common conditions in real-world annotation projects.
Alpha is computed as one minus the ratio of observed disagreement to expected disagreement under chance. The key difference from kappa is that alpha uses a unified disagreement function that can be adapted to the measurement scale: disagreement between adjacent ordinal categories is weighted less than disagreement between extreme ones, interval distances are respected for continuous ratings, and nominal disagreement treats all mismatches equally.
Krippendorff's alpha is a dataset-level metric used to quantify inter-rater reliability. Unlike other agreement measures, it is particularly useful for messy, real-world datasets where not all annotators rate every item.
When to use which:
Use Cohen's kappa for exactly two annotators, nominal categories, pairwise reliability checks. Use Fleiss's kappa when more than two annotators have all labelled every item. Use Krippendorff's alpha for everything else: large pools with partial overlap, ordinal or continuous scales, missing data, or when you need a single metric comparable across task structures.
In production multilingual annotation, alpha is almost always the right choice because annotator assignment is rarely uniform.
How do you read the numbers?
Carefully, with reference to the task. The thresholds come from Landis and Koch (1977) for kappa; Krippendorff is more conservative for alpha.
For Cohen's kappa: 0.0 to 0.20 is slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, and 0.81 to 1.00 almost perfect agreement. By an alternative convention from Randolph (2008), below 0.40 is poor, 0.40 to 0.75 intermediate to good, and 0.75 and above excellent.
Krippendorff recommends treating data below an alpha of 0.667 with caution and data below 0.800 as unreliable for drawing conclusions, though acceptable levels depend on the task.
A kappa of 0.60 means different things in different tasks. For toxicity classification on ambiguous edge cases, where expert linguists disagree on the same examples, 0.60 may reflect genuine task difficulty rather than annotator failure. For clear binary categories, 0.60 would be cause for concern.
Real annotation reports pair the headline figure with per-category breakdown and confusion matrices, because these show where disagreement lives rather than only how much there is.
What does disagreement actually tell you?
It tells you where the task is ambiguous, where the guidelines are unclear, and sometimes where the categories are wrong. Disagreement is information, not only failure.
Analysis of annotation quality practices in over 100 NLP dataset papers found that quality assurance is routinely underreported. Most papers report a headline IAA figure without the per-category breakdown that would reveal which labels drive disagreement. This hides the information that would actually improve the dataset.
Three things systematic disagreement usually signals:
Guideline ambiguity. Consistent disagreement on the same category means the instruction is unclear. The fix is a guideline update and re-annotation of disputed items, not a performance conversation.
Category design problems. Persistent disagreement can mean a label conflates two distinct things, or a boundary has been drawn in the wrong place.
Genuine task difficulty. For subjective tasks like emotion, irony and sarcasm, IAA measures how contested the task is rather than annotation quality. Preserving disagreement in the dataset may be more useful than forcing consensus.
In multilingual annotation, disagreement that is high in one language and low in another can indicate a category that translates poorly or a cultural concept that does not map cleanly. A single aggregate IAA figure will not show this; per-language breakdown, as Lifewood's human-in-the-loop model produces, is what surfaces it.
Key takeaways
- Raw percentage agreement is inflated by chance; kappa and alpha correct for this.
- Cohen's kappa handles exactly two annotators on nominal categories, subtracting the expected chance agreement from observed agreement.
- The kappa paradox: skewed label distributions produce low kappa even when annotators are consistent, because expected chance agreement is already high.
- Krippendorff's alpha handles any number of annotators, tolerates missing data and works across nominal, ordinal, interval and ratio scales.
- Landis and Koch (1977): 0.0 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, 0.81 to 1.00 almost perfect.
- Krippendorff recommends 0.800 as the floor for reliable conclusions and treating data below 0.667 with caution.
- The same score means different things in tasks of different difficulty. Always report per-category breakdown alongside the headline figure.
- Systematic disagreement diagnoses guideline ambiguity, category design problems or genuine perceptual difficulty; preserving it may be more useful than forcing consensus.
- Per-language IAA breakdown can reveal categories that translate poorly across languages.
- Over 100 NLP dataset papers were found to routinely under-report quality assurance information.
Sources and further reading
- Claru, "Inter-Annotator Agreement: Metrics and Best Practices", on Cohen's kappa history, Krippendorff alpha history and the Landis and Koch interpretation scale
- Label Studio, "Krippendorff's Alpha for Annotation Agreement"
- Artstein, "Inter-annotator Agreement: A Survey", on mathematical properties of kappa and alpha
- arXiv, "Selecting the Right Inter-annotator Agreement Metric", on reporting practices and per-category analysis
- arXiv, "Temporal Simultaneity Predicts Annotation Quality in Sentiment Corpora", citing Klie et al. (2024) on underreporting in NLP papers and Randolph's kappa thresholds
- arXiv, "Do We Still Need Humans in the Loop?", on variable-annotator-subset use of Krippendorff's alpha
- Lifewood, annotation services and human-in-the-loop quality assurance