Inter-Annotator Agreement: Cohen's Kappa, Krippendorff's Alpha and What the Numbers Mean

Imagine two annotators labelling a binary classification task where 90% of items belong to Class A. Both annotators can score 81% raw agreement simply by guessing Class A every time, without ever engaging with the actual data. The raw percentage looks reasonable. The agreement is meaningless.

This is the foundational problem Cohen's kappa was designed to address. Jacob Cohen introduced the kappa statistic in 1960 specifically to measure agreement between two psychiatric diagnosticians rating patient symptoms, correcting for the base rate of chance agreement that inflated raw percent agreement scores.

The same inflation problem appears in multilingual annotation. A dataset where one label category dominates, which is common in sentiment, toxicity and intent classification tasks, will produce high raw agreement even when annotators are applying the labels inconsistently. Teams building multilingual training data for LLMs run into this constantly, since label distributions rarely stay balanced once a corpus spans many languages. The chance-corrected metrics are the ones that expose this.

Cohen's kappa is the ratio of observed agreement above chance to the maximum possible agreement above chance between exactly two annotators. It answers: of the agreement that could not have happened by chance, how much actually did?

The formula is straightforward. Kappa equals observed agreement minus expected agreement, divided by one minus expected agreement. Expected agreement is calculated from the marginal distributions of each annotator's labels: if Annotator A labels 60% of items positive and Annotator B labels 55% positive, the expected chance agreement is the probability that two independent raters would land on the same label under those distributions.

Cohen's kappa yields a value from minus 1 (perfect disagreement) to 1 (perfect agreement), with 0 indicating chance-level agreement.

Key limitations worth knowing:

Cohen's kappa is for exactly two annotators. Fleiss's kappa extends this to multiple annotators using the same formula structure, but it requires every annotator to label every item. Krippendorff's alpha is used when Fleiss's kappa is not applicable, for example for variable annotator subsets where not every item is annotated by the same people.

Kappa is sensitive to label distribution. This is the kappa paradox: the pattern in which a skewed label distribution pushes kappa down even though annotators are being consistent, because the expected agreement is very high and there is little room for observed agreement to exceed it. A kappa of 0.40 on a heavily skewed task can represent the same underlying consistency as a kappa of 0.70 on a balanced one.

It measures nominal categories only in its standard form. Ordinal scales, continuous ratings and spans require different treatment, which is one reason large text annotation for NLP programmes often move to alpha once tasks mix category types.

Krippendorff's alpha is a chance-corrected reliability metric that generalises across any number of annotators, tolerates missing data, and works on nominal, ordinal, interval and ratio scales. In large annotation pipelines it is almost always the right choice.

Klaus Krippendorff developed alpha in 1970 for content analysis in communication research, where multiple coders categorised media content. His 2004 formalisation established alpha as the most general-purpose reliability metric, handling arbitrary numbers of coders, missing data, and multiple measurement levels, all common conditions in real-world annotation projects.

Alpha is computed as one minus the ratio of observed disagreement to expected disagreement under chance. The key difference from kappa is that alpha uses a unified disagreement function that can be adapted to the measurement scale: disagreement between adjacent ordinal categories is weighted less than disagreement between extreme ones, interval distances are respected for continuous ratings, and nominal disagreement treats all mismatches equally.

Alpha is a dataset-level metric, and it is particularly useful for messy, real-world datasets where not all annotators rate every item — the situation most human-in-the-loop annotation pipelines actually run in once work is distributed across a large contributor pool.

Use Cohen's kappa for exactly two annotators, nominal categories, pairwise reliability checks. Use Fleiss's kappa when more than two annotators have all labelled every item. Use Krippendorff's alpha for everything else: large pools with partial overlap, ordinal or continuous scales, missing data, or when a single metric needs to be comparable across task structures. In production multilingual annotation, alpha is almost always the right choice because annotator assignment is rarely uniform.

For Cohen's kappa: 0.0 to 0.20 is slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, and 0.81 to 1.00 almost perfect agreement. By an alternative convention from Randolph (2008), below 0.40 is poor, 0.40 to 0.75 intermediate to good, and 0.75 and above excellent.

Krippendorff recommends treating data below an alpha of 0.667 with caution and data below 0.800 as unreliable for drawing conclusions, though acceptable levels depend on the task.

A kappa of 0.60 means different things in different tasks. For toxicity classification on ambiguous edge cases, where expert linguists disagree on the same examples, 0.60 may reflect genuine task difficulty rather than annotator failure. For clear binary categories, 0.60 would be cause for concern. This is why an AI data validation process should report per-category breakdowns and confusion matrices alongside the headline figure, since these show where disagreement lives rather than only how much there is.

Analysis of annotation quality practices in over 100 NLP dataset papers found that quality assurance is routinely underreported. Most papers report a headline IAA figure without the per-category breakdown that would reveal which labels drive disagreement. This hides the information that would actually improve the dataset.

Three things systematic disagreement usually signals:

Guideline ambiguity. Consistent disagreement on the same category means the instruction is unclear. The fix is a guideline update and re-annotation of disputed items, not a performance conversation, a pattern covered in more detail in how to write annotation guidelines annotators actually follow.

Category design problems. Persistent disagreement can mean a label conflates two distinct things, or a boundary has been drawn in the wrong place.

Genuine task difficulty. For subjective tasks like emotion, irony and sarcasm, IAA measures how contested the task is rather than annotation quality. Preserving disagreement in the dataset may be more useful than forcing consensus.

In multilingual annotation, disagreement that is high in one language and low in another can indicate a category that translates poorly or a cultural concept that does not map cleanly. A single aggregate IAA figure will not show this; per-language breakdown, produced through managed multilingual data collection, is what surfaces it. Lifewood's quality process pairs a 95%+ inter-annotator agreement threshold, measured against a customer-approved gold set, with two independent review passes and timestamped approval records, so a drop in agreement on a specific category or language is visible before delivery rather than after.