Text Annotation for NLP: NER, Sentiment, Intent and Relation Labelling

Short answer. Four task types dominate text annotation, and each fails differently. NER breaks on span boundaries and fuzzy categories. Sentiment breaks on sarcasm, mixed opinion and domain context. Intent breaks on overlapping classes and messy conversational speech. Relation labelling breaks on implicit and directional links. The good news is that these are solved problems where the discipline exists: with proper training, tooling and iterated guidelines, high agreement between annotators is demonstrably achievable — and one study of German relation annotation reached a Cohen's κ of 0.92.

What are the four task types?

Different label structures for different jobs — and choosing the right structure at the start is what makes the pipeline smooth and the model accurate.

There is no generic solution for text annotation. NER is token-level work, usually tagged with a Begin-Inside-Outside (BIO) scheme, and it underpins information retrieval, question answering and event extraction. Sentiment assigns polarity or, in richer schemes, discrete emotions such as joy, anger or frustration. Intent classifies what the speaker wants, and drives virtual assistants and support routing. Relation labelling tags free-form spans and the links between them, and is the basis of knowledge-graph construction.

The encouraging finding across the literature is that difficulty is not destiny. Research on annotating neurologic signs in electronic health records — a domain where prior studies had suggested agreement would be low — found inter-rater agreement between three raters was high for both text span and category label after training on the process, the tool and the supporting ontology. The authors' conclusion is the one worth carrying into any project: high levels of agreement between human annotators are possible with appropriate training and annotation tools.

Where does each one trip up annotators?

Predictably, and differently — which is why one set of guidelines cannot serve all four.

The four task types and their characteristic failure points TASK WHERE ANNOTATORS TRIP UP THE FIX VERDICT NER Span boundaries, nested entities, partial-token selection, and catch-all classes — PER, LOC and ORG reliably outscore MISC on agreement Token-wise span control; drop or split the MISC bucket; per-entity F1 to find weak types Fixable with schema design SENTIMENT Sarcasm, mixed opinion, cultural difference, and the neutral boundary; medical sentiment is not retail sentiment Explicit positive/negative/neutral definitions with edge-case examples; domain-specific guidance Lowest agreement of the four INTENT Overlapping or near-duplicate classes; conversational speech adds disfluencies, interruptions and implicit references to slot-filling Mutually exclusive class definitions; a flag-and-escalate route for genuine ambiguity Schema discipline wins RELATION Implicit versus explicit links, direction, and the silent dominance of neutral or "no relation" cases in long documents Rule on implicit relations up front; annotate negatives explicitly; use F1 over relations, not κ Hardest, highest value Annotation quality sets the ceiling: with annotator agreement at 80%, a model reaching 83% is already performing at roughly human level.

Relation labelling deserves particular attention because its difficulty is uneven within a single schema. In a multilingual relation-extraction study, automatic labels were checked against 2,000 manually annotated sentences: some relations were near-perfect while others collapsed entirely.

Two lessons follow. First, a macro F1 of 0.79 across that schema conceals a relation performing at 0.33; always break quality metrics down by class. Second, the same study's human annotators reached a Cohen's κ of 0.92, with disagreements discussed and resolved case by case — strong evidence that the hard part is guideline clarity and adjudication process, not human capability.

Sentiment sits at the other end. Benchmark work using Krippendorff's alpha found sentiment tasks showed lower agreement than named entity recognition tasks in the same corpora, and the practical reason is that neutral is rarely a clean category. In one analytical-text study, annotators asked to mark only positive or negative relations were internally classifying a third, neutral class that then dominated the data — a schema problem masquerading as an annotator problem.

Which metric should you use for which task?

Match the metric to the label structure, or your quality number will mislead you.

USE THE RIGHT MEASURE AVOID THESE TRAPS

Cohen's κ — two annotators, categorical labels:

sentiment, intent

Using Cohen's κ for NER — it needs negative cases and badly underestimates agreement
Krippendorff's α — multiple annotators, missing data, ordinal or hierarchical labels
Reporting one averaged score across all classes
Token-level pairwise F1 (excluding O) — the right IAA measure for NER
F1 over relations — for relation extraction, where κ does not fit
Gwet's AC2 — when categories are heavily imbalanced Clinical datasets often require α > 0.90 before release; general targets sit near 80% agreement.
Wrong α setting for the data type, which can skew the result either way
Treating a low score as an annotator failure rather than a guideline gap
Measuring once instead of continuously on a sample In one Tweebank NER corpus, κ read 0.347 while the appropriate F1 measure showed agreement of 0.71.

Two operational points make the difference in practice. Continuous monitoring of agreement on a subset surfaces ambiguities in the guidelines early, which is when they are cheap to fix; guidelines almost always need refinement, and the biomedical gold-standard literature shows acceptable agreement being reached across multiple iterations by tightening rules and establishing semantic equivalence criteria. And per-class F1 for NER pinpoints exactly which entity types annotators struggle with, turning a vague quality problem into a targeted training session.

Multilingual work adds a layer that no metric captures. Sarcasm, politeness and negation behave differently across languages, and a guideline written in English rarely transfers cleanly. Native linguists, pilot batches per language, and per-language agreement tracking are exactly the human-in-the-loop discipline Lifewood applies across 50+ languages and dialects.

What should you fix first?

In this order, because each step raises the ceiling for the next.

Run a pilot batch before full production. Two to three annotators on the same data, target 80%+ agreement, and treat the disagreements as your guideline backlog.

Write edge cases into the guidelines, not just definitions. Clear boundaries for positive, negative and neutral with explicit examples measurably improve agreement.

Pick the metric that fits the label structure. Token-level F1 for NER, κ or α for classification, F1 over relations for relation work.

Break every quality score down by class. A macro score of 0.79 can hide a relation type at 0.33 and an entity type in free fall.

Match annotators to the domain. Expert pairs cleared the acceptability threshold in the ARFBench data while non-expert pairs did not.

Pre-label with a model, then have humans review. LLM pre-labelling works for basic sentiment and classification at low cost per example, but struggles with domain expertise and subjective judgement — so keep the two-pass workflow of initial annotation plus expert validation.

Re-annotate the grey zone, not the whole set. Pulling the trickiest 5–10% back through QA under refined rules is far cheaper than a full pass.

Run per-language pilots and agreement checks. Guidelines written in one language do not transfer unexamined to another.

Key takeaways

Four task types dominate text annotation — NER, sentiment, intent and relation labelling — and each has its own characteristic failure point.
High agreement is achievable: EHR annotation research found high inter-rater agreement on span and category after proper training, and a German relation study reached κ = 0.92.
NER trips on span boundaries, nested entities and catch-all classes; PER, LOC and ORG outperform MISC on agreement.
Sentiment shows lower agreement than NER in the same corpora, mainly because neutral boundaries and sarcasm are poorly specified.
Relation quality varies hugely within one schema — from F1 0.99 on birthdate to 0.33 on deathplace — so report per-class scores.
Use token-level F1 for NER, κ or Krippendorff's α for classification, F1 over relations for relation work; κ underestimated one NER corpus at 0.347 against a true 0.71.
• LLM pre-labelling plus human review is effective for simple tasks but struggles with domain expertise and subjective judgement.

Sources and further reading

Frequently asked questions

Around 80% agreement is a common general target, with κ or α of roughly 0.667 treated as tentatively acceptable and clinical datasets often requiring α above 0.90 before release. Set the bar by the risk profile of the application.

Because κ needs a count of negative cases and NER is a sequence-tagging task. Token-level pairwise F1 calculated without the O label is the established alternative.

Partly. They work well for basic sentiment and simple classification and are effective for fast pre-labelling, but they struggle with domain expertise, subjective judgement and consistency — so the recommended pattern is LLM pre-labels followed by human review.

Have an AI or visibility project in mind?

From AI evaluation and human-in-the-loop review to GEO and AEO strategy, our team can help you deploy with confidence and get found in the AI search era.

Talk to our team