Short answer. Four task types dominate text annotation, and each fails differently. NER breaks on span boundaries and fuzzy categories. Sentiment breaks on sarcasm, mixed opinion and domain context. Intent breaks on overlapping classes and messy conversational speech. Relation labelling breaks on implicit and directional links. The good news is that these are solved problems where the discipline exists: with proper training, tooling and iterated guidelines, high agreement between annotators is demonstrably achievable — and one study of German relation annotation reached a Cohen's κ of 0.92.
Key takeaways
- Four task types dominate text annotation — NER, sentiment, intent and relation labelling — and each has its own characteristic failure point.
- High agreement is achievable: EHR annotation research found high inter-rater agreement on span and category after proper training, and a German relation study reached κ = 0.92.
- NER trips on span boundaries, nested entities and catch-all classes; PER, LOC and ORG outperform MISC on agreement.
- Sentiment shows lower agreement than NER in the same corpora, mainly because neutral boundaries and sarcasm are poorly specified.
- Use token-level F1 for NER, κ or Krippendorff's α for classification, and F1 over relations for relation work; κ underestimated one NER corpus at 0.347 against a true 0.71.
What are the four task types?
There is no generic solution for text annotation: NER, sentiment, intent and relation labelling each use a different label structure, and choosing the right one at the start is what keeps the pipeline smooth and the model accurate.
Named entity recognition (NER) is token-level work, usually tagged with a Begin-Inside-Outside (BIO) scheme, and it underpins information retrieval, question answering and event extraction. Sentiment assigns polarity or, in richer schemes, discrete emotions such as joy, anger or frustration. Intent classifies what the speaker wants, and drives virtual assistants and support routing. Relation labelling tags free-form spans and the links between them, and is the basis of knowledge-graph construction.
The encouraging finding across the literature is that difficulty is not destiny. Research on annotating neurologic signs in electronic health records — a domain where prior studies had suggested agreement would be low — found inter-rater agreement between three raters was high for both text span and category label after training on the process, the tool and the supporting ontology. The authors' conclusion is the one worth carrying into any project: high levels of agreement between human annotators are possible with appropriate training and annotation tools.
Where does each one trip up annotators?
Predictably, and differently — which is why one set of guidelines cannot serve all four task types.
| Task | Where annotators trip up | The fix | Verdict |
|---|---|---|---|
| NER | Span boundaries, nested entities, partial-token selection, and catch-all classes — PER, LOC and ORG reliably outscore MISC on agreement | Token-wise span control; drop or split the MISC bucket; per-entity F1 to find weak types | Fixable with schema design |
| Sentiment | Sarcasm, mixed opinion, cultural difference, and the neutral boundary; medical sentiment is not retail sentiment | Explicit positive/negative/neutral definitions with edge-case examples; domain-specific guidance | Lowest agreement of the four |
| Intent | Overlapping or near-duplicate classes; conversational speech adds disfluencies, interruptions and implicit references to slot-filling | Mutually exclusive class definitions; a flag-and-escalate route for genuine ambiguity | Schema discipline wins |
| Relation | Implicit versus explicit links, direction, and the silent dominance of neutral or "no relation" cases in long documents | Rule on implicit relations up front; annotate negatives explicitly; use F1 over relations, not κ | Hardest, highest value |
Annotation quality sets the ceiling: with annotator agreement at 80%, a model reaching 83% is already performing at roughly human level.
Relation labelling deserves particular attention because its difficulty is uneven within a single schema. In a multilingual relation-extraction study, automatic labels were checked against 2,000 manually annotated sentences: some relations were near-perfect while others collapsed entirely. Two lessons follow. First, a macro F1 of 0.79 across that schema conceals a relation type performing at 0.33, so quality metrics should always be broken down by class. Second, the same study's human annotators reached a Cohen's κ of 0.92, with disagreements discussed and resolved case by case — strong evidence that the hard part is guideline clarity and adjudication process, not human capability.
Sentiment sits at the other end. Benchmark work using Krippendorff's alpha found sentiment tasks showed lower agreement than named entity recognition tasks in the same corpora, and the practical reason is that neutral is rarely a clean category. In one analytical-text study, annotators asked to mark only positive or negative relations were internally classifying a third, neutral class that then dominated the data — a schema problem masquerading as an annotator problem.
Which metric should you use for which task?
Match the metric to the label structure, or your quality number will mislead you: a single agreement score cannot serve categorical labels, span tags and relation links equally well.
Cohen's κ is a two-annotator agreement statistic for categorical labels, and it fits sentiment and intent well. Krippendorff's α extends that idea to multiple annotators, missing data, and ordinal or hierarchical labels. Neither is the right inter-annotator agreement measure for every task, which is the trap most teams fall into.
| Metric and when to use it | Avoid this trap |
|---|---|
| Cohen's κ — two annotators, categorical labels: sentiment, intent | Using Cohen's κ for NER — it needs negative cases and badly underestimates agreement |
| Krippendorff's α — multiple annotators, missing data, ordinal or hierarchical labels | Reporting one averaged score across all classes |
| Token-level pairwise F1 (excluding the "O" label) — the right IAA measure for NER | Wrong α setting for the data type, which can skew the result either way |
| F1 over relations — for relation extraction, where κ does not fit | Treating a low score as an annotator failure rather than a guideline gap |
| Gwet's AC2 — when categories are heavily imbalanced | Measuring once instead of continuously on a sample |
Clinical datasets often require α above 0.90 before release; general targets sit near 80% agreement. The gap between metrics is not academic: in one Tweebank NER corpus, κ read 0.347 while the appropriate F1 measure showed agreement of 0.71 on the same annotations.
Two operational points make the difference in practice. Continuous monitoring of agreement on a subset surfaces ambiguities in the guidelines early, which is when they are cheap to fix, and the biomedical gold-standard literature shows acceptable agreement being reached across multiple iterations by tightening rules and establishing semantic equivalence criteria. Per-class F1 for NER pinpoints exactly which entity types annotators struggle with, turning a vague quality problem into a targeted training session.
Multilingual work adds a layer that no metric captures. Sarcasm, politeness and negation behave differently across languages, and a guideline written in English rarely transfers cleanly. Native linguists, pilot batches per language, and per-language agreement tracking are exactly the human-in-the-loop discipline Lifewood applies across 50+ languages and dialects, drawing on the same multilingual data collection network used for speech and text programmes.
What should you fix first?
Fix the pipeline in this order, because each step raises the ceiling for the next.
Run a pilot batch before full production, with two to three annotators on the same data, a target of 80%+ agreement, and the disagreements treated as a guideline backlog. Write edge cases into the annotation guidelines, not just definitions — clear boundaries for positive, negative and neutral with explicit examples measurably improve agreement. Pick the metric that fits the label structure: token-level F1 for NER, κ or α for classification, F1 over relations for relation work. Break every quality score down by class, since a macro score of 0.79 can hide a relation type at 0.33 and an entity type in free fall.
Match annotators to the domain — expert pairs cleared the acceptability threshold in the ARFBench data while non-expert pairs did not. Pre-label with a model, then have humans review: LLM pre-labelling and active learning works for basic sentiment and classification at low cost per example, but struggles with domain expertise and subjective judgement, so the two-pass workflow of initial annotation plus expert validation stays the safer default. Re-annotate the grey zone rather than the whole set — pulling the trickiest 5–10% back through a gold-set audit under refined rules is far cheaper than a full pass. Finally, run per-language pilots and agreement checks, since guidelines written in one language do not transfer unexamined to another.
Teams sourcing this work at scale can compare providers against the criteria in a buyer's guide to data annotation companies before committing to a vendor or an in-house build.