Short answer. Multimodal annotation is less like data entry than like writing law: drawing a box or transcribing a clip is fast, but deciding what a label means at the boundary — an occluded object, a reflection, where a gesture begins — is not. Those decisions live in a guideline document, and dataset quality is that document's quality plus how consistently it is applied — why cost tracks ambiguity density over volume, and quality is measured with inter-annotator agreement, not raw accuracy.
Most annotation buyers ask for a price per unit and are surprised when quotes vary by an order of magnitude. The variance comes from the parts of the specification usually left blank: which task, against which taxonomy, at what overlap rate, adjudicated by whom, and with what tolerance for items nobody can label confidently.
Key takeaways
- Cost per annotated unit varies by more than an order of magnitude across modalities; the driver is ambiguity density and the required agreement level, not raw volume.
- A dataset's real deliverable is its guideline document — a definition per class, edge-case rulings with examples, and a documented rule for the unsure case.
- Because ground truth usually does not exist, quality is established through overlap, chance-corrected agreement statistics and senior adjudication rather than a single accuracy number.
- Model-assisted labelling raises throughput but can anchor annotators on a pre-labelling model's own blind spots, especially on ambiguous items.
- Multimodal work adds alignment requirements on top of single-modality labelling: a canonical timeline, shared object identifiers, and documented sensor calibration.
What does annotation actually involve, by modality?
The word covers a wide range of tasks with different cost profiles and different failure modes. Naming which one you need, precisely, is the first step in getting a comparable quote from anyone.
| Modality | Common tasks | Where the difficulty actually is |
|---|---|---|
| Image | Classification, bounding boxes, polygons, semantic and instance segmentation, keypoints | Boundary definition — occlusion, truncation, crowds, reflections, and what counts as one instance |
| Video | Object tracking, action segmentation, temporal event boundaries, re-identification | Temporal consistency. The same object must keep its identity across frames, including through occlusion |
| Audio | Transcription, speaker diarisation, event tagging, emotion and intent labels | Overlapping speech, accents and dialects, noise, and the inherent subjectivity of affect labels |
| Text | Entity extraction, intent, sentiment, relation extraction, ranking and preference data | Definitional edge cases and annotator cultural framing, especially in judgement tasks |
| Sensor fusion | 3D boxes across camera, LiDAR and radar; map and telemetry association | Calibration and coordinate systems. One bad extrinsic shifts every box, however well the annotator followed the guideline |
| Cross-modal | Caption alignment, grounding, video-question pairs, preference comparison | Two taxonomies must agree, so ambiguity in either propagates into the pairing |
Cost per unit varies by more than an order of magnitude down that table. A quote that names none of the task type, the taxonomy size or the expected edge-case density is a placeholder rather than a quote.
Why is the guideline document the real deliverable?
A dataset is an operationalised definition, and the document that defines it matters more than the labeling tool used to apply it. "Label all vehicles" is not a definition — it is a topic. A definition says whether a bicycle is a vehicle, whether a vehicle behind a fence at twenty per cent visibility is annotated, whether a reflection in a window is an instance, and what an annotator should do when genuinely unsure.
Every one of those questions gets answered by every annotator, whether or not the guideline answers it. If the document is silent, each annotator answers it privately and differently — inconsistency that is invisible in a spot check and highly visible to a model trained on the result. The practical steps for writing annotation guidelines that hold up at scale follow the same pattern below:
- A definition per class, written as an inclusion and exclusion test rather than a description.
- Explicit edge-case rulings, each with an example image or clip. This section grows throughout the project and is the highest-value artefact it produces.
- A rule for the unsure case — a skip or flag path, with adjudication. Forcing a guess on ambiguous items manufactures noise and hides the ambiguity from the people who could resolve it.
- Worked examples, including near-misses. Positive examples teach the centre of the category; near-misses teach the boundary, where all the disagreement lives.
- Versioning. When a ruling changes, the affected earlier work has to be identified and re-adjudicated. Undocumented drift is how a dataset ends up internally inconsistent by construction.
The cheapest available diagnostic: take twenty genuinely difficult items and have three annotators label them independently against the current guideline. Wherever they disagree, the guideline is under-specified. The exercise takes an hour and predicts dataset quality better than any amount of downstream QA.
How is quality measured when there is no ground truth?
There usually is none — if there were, the annotation would be unnecessary. So quality is established through agreement and adjudication rather than through an accuracy score against a known answer.
- Overlap a defined share of the work. A stated percentage of items labelled independently by two or more annotators — the raw material for every quality statement that follows, and something that has to be budgeted from the start.
- Compute chance-corrected agreement. Cohen's kappa or an appropriate variant for categorical labels; raw percentage agreement flatters tasks with imbalanced classes and should not be reported alone.
- Interpret against a stated scale, and say which one. The Landis and Koch scale (Biometrics, 1977) — 0.61–0.80 substantial, above 0.80 almost perfect — remains the common reference, and its authors presented it as arbitrary benchmarks rather than statistical thresholds. Treat it as a convention, and name it.
- Adjudicate disagreements with a senior reviewer. Every disagreement is either a genuine ambiguity, which becomes a guideline ruling, or an annotator error, which becomes feedback. Sorting them is what improves the dataset over time.
- Audit with a gold set the annotators cannot identify. Items with adjudicated answers, mixed into ordinary work, measuring sustained performance rather than performance during a known evaluation.
- Track agreement over time and per annotator. A falling trend usually means fatigue or drift; a single divergent annotator usually means a correctable misunderstanding. Both are visible only if measurement is continuous.
Where that measurement becomes a contractual threshold — which metric, what number, sampled how, what happens when a batch fails — is covered separately in setting an annotation accuracy standard and SLA. The point here is narrower: without overlap and adjudication in the budget, there is no quality statement to put in an SLA at all.
What breaks when the modalities have to agree?
Single-modality work can be judged inside one signal. Multimodal work adds relationships, and each one is a new place for two correct-looking labels to describe different things.
A spoken phrase aligning with a video event, a caption describing the correct region, a camera box and a LiDAR box referring to one physical object — the schema needs cross-modal identifiers, timing rules, synchronisation tolerances, and a precedence rule for when signals disagree.
- Define a canonical timeline and state how each modality maps onto it, with the tolerance in frames or milliseconds written down.
- Carry shared IDs. Object identity has to be the same string in every stream, or the pairing cannot be checked automatically.
- Document coordinate systems and calibration for sensor fusion, and re-verify them per capture session rather than per project — the same discipline that autonomous driving annotation programmes depend on.
- Say what a paired annotation describes — the whole scene, a region, an event, or a moment. Ambiguity here produces pairs that are individually correct and jointly meaningless.
Errors propagate, which is why a single blended accuracy figure hides the cause. A bad transcript makes an image–text pair look mismatched; a calibration error shifts every 3D box even where annotators followed the guideline perfectly. Report label correctness, temporal alignment, spatial alignment, completeness and semantic consistency separately, and pilot the pipeline end to end before scaling — multimodal rework has to be undone across every file already processed.
Does model-assisted labelling introduce bias?
Yes. Annotators presented with a plausible model suggestion accept it more often than they would have produced the label themselves, and the effect is strongest exactly where a dataset most needs independent human judgment: on ambiguous items, where a confident-looking box resolves the annotator's uncertainty in the model's favour.
Pre-labelling with a model and having annotators correct the output is now standard, and the throughput gain is genuine. So is the cost. The consequence is that model-assisted labelling can encode the pre-labelling model's blind spots and then train a model that inherits them — with the agreement statistics healthy throughout, because annotators agree with each other about accepting the same suggestions.
- Keep unassisted control batches. The divergence between them and the assisted stream is the measurement of the anchoring effect.
- Suppress low-confidence suggestions. Where the model is unsure, showing nothing produces better labels than showing a guess.
- Audit the accepted-without-change rate. A very high rate is a warning sign, not an efficiency achievement.
- Never pre-label with the model being evaluated. It manufactures agreement between the dataset and the system it is meant to test.
What actually drives the cost?
Four factors, only one of which is volume.
- Ambiguity density. The share of items needing adjudication. Clean data is cheap at any volume; ambiguous data is expensive at any volume.
- Taxonomy size and depth. Deep hierarchies multiply the boundary decisions an annotator makes per item.
- Required agreement level. Higher targets mean more overlap, more adjudication and more senior review time — the lever buyers control most directly and understand least.
- Specialist knowledge. Medical, legal, industrial or language-specific tasks need qualified annotators, which changes the labour pool and the price.
Adjudication load = Items sent to a senior reviewer ÷ Items labelled
Track that ratio from the first batch. It is the earliest reliable signal of whether a project is priced correctly, and it moves long before any agreement statistic does. For scale rather than precision: Grand View Research estimates the global data collection and labelling market at USD 6.3 billion for 2026, growing at a 28.4% compound annual rate through 2030 — a research-firm estimate, not an audited total.
How does Lifewood approach multimodal annotation at scale?
Lifewood runs annotation programmes on versioned guidelines, overlap budgeted from the start, chance-corrected agreement measured continuously, and adjudication by senior reviewers, with dual-layer human-in-the-loop QA held to a 95%+ accuracy threshold.
For multimodal and multilingual work the constraint is who is available to judge: 100+ languages and 40+ delivery centres across 30+ countries mean audio and text are adjudicated by people who hear when something is subtly wrong, rather than by a reviewer working from a translated guideline. Lifewood's data-annotation practice traces back to 2004, giving it over two decades of operating history in this work. That regional coverage is also what sensor-fusion QA for autonomous driving depends on, and it underpins broader AI data validation work across modalities.