Short answer. Multimodal annotation is less like data entry than like writing law. Drawing the box, marking the span and transcribing the clip is fast and largely solved by tooling. The expensive part is deciding what the labels mean at the boundaries — whether a partially occluded object counts, whether a reflection is an instance, where a gesture begins — and, once two or more modalities are involved, keeping time, identity and meaning aligned across them. Those decisions live in a guideline document, and the quality of a dataset is essentially the quality of that document plus the consistency with which annotators apply it. That is why the work is measured with inter-annotator agreement rather than a raw accuracy figure, and why cost tracks ambiguity density far more closely than it tracks volume.
Most annotation buyers ask for a price per unit and are surprised when quotes vary by an order of magnitude. The variance comes from the parts of the specification usually left blank: which task, against which taxonomy, at what overlap rate, adjudicated by whom, and with what tolerance for items nobody can label confidently.
What does annotation actually involve, by modality?
The word covers a wide range of tasks with different cost profiles and different failure modes. Naming which one you need, precisely, is the first step in getting a comparable quote from anyone.
| Modality | Common tasks | Where the difficulty actually is |
|---|---|---|
| Image | Classification, bounding boxes, polygons, semantic and instance segmentation, keypoints | Boundary definition — occlusion, truncation, crowds, reflections, and what counts as one instance |
| Video | Object tracking, action segmentation, temporal event boundaries, re-identification | Temporal consistency. The same object must keep its identity across frames, including through occlusion |
| Audio | Transcription, speaker diarisation, event tagging, emotion and intent labels | Overlapping speech, accents and dialects, noise, and the inherent subjectivity of affect labels |
| Text | Entity extraction, intent, sentiment, relation extraction, ranking and preference data | Definitional edge cases and annotator cultural framing, especially in judgement tasks |
| Sensor fusion | 3D boxes across camera, LiDAR and radar; map and telemetry association | Calibration and coordinate systems. One bad extrinsic shifts every box, however well the annotator followed the guideline |
| Cross-modal | Caption alignment, grounding, video-question pairs, preference comparison | Two taxonomies must agree, so ambiguity in either propagates into the pairing |
Cost per unit varies by more than an order of magnitude down that table. A quote that names none of the task type, the taxonomy size or the expected edge-case density is a placeholder rather than a quote.
Why is the guideline document the real deliverable?
A dataset is an operationalised definition. "Label all vehicles" is not a definition — it is a topic. A definition says whether a bicycle is a vehicle, whether a vehicle behind a fence at twenty per cent visibility is annotated, whether a reflection in a window is an instance, and what an annotator should do when genuinely unsure.
Every one of those questions gets answered by every annotator, whether or not the guideline answers it. If the document is silent, each annotator answers it privately and differently — inconsistency that is invisible in a spot check and highly visible to a model trained on the result.
- A definition per class, written as an inclusion and exclusion test rather than a description.
- Explicit edge-case rulings, each with an example image or clip. This section grows throughout the project and is the highest-value artefact it produces.
- A rule for the unsure case — a skip or flag path, with adjudication. Forcing a guess on ambiguous items manufactures noise and hides the ambiguity from the people who could resolve it.
- Worked examples, including near-misses. Positive examples teach the centre of the category; near-misses teach the boundary, where all the disagreement lives.
- Versioning. When a ruling changes, the affected earlier work has to be identified and re-adjudicated. Undocumented drift is how a dataset ends up internally inconsistent by construction.
The cheapest available diagnostic: take twenty genuinely difficult items and have three annotators label them independently against the current guideline. Wherever they disagree, the guideline is under-specified. The exercise takes an hour and predicts dataset quality better than any amount of downstream QA.
How is quality measured when there is no ground truth?
There usually is none — if there were, the annotation would be unnecessary. So quality is established through agreement and adjudication rather than through an accuracy score against a known answer.
- Overlap a defined share of the work. A stated percentage of items labelled independently by two or more annotators — the raw material for every quality statement that follows, and something that has to be budgeted from the start.
- Compute chance-corrected agreement. Cohen's kappa for two annotators on categorical labels, or an appropriate variant otherwise. Raw percentage agreement flatters tasks with imbalanced classes and should not be reported alone.
- Interpret against a stated scale, and say which one. The Landis and Koch scale (Biometrics, 1977) — 0.61–0.80 substantial, above 0.80 almost perfect — remains the common reference, and its authors presented it as arbitrary benchmarks rather than statistical thresholds. Treat it as a convention, and name it.
- Adjudicate disagreements with a senior reviewer. Every disagreement is either a genuine ambiguity, which becomes a guideline ruling, or an annotator error, which becomes feedback. Sorting them is what improves the dataset over time.
- Audit with a gold set the annotators cannot identify. Items with adjudicated answers, mixed into ordinary work, measuring sustained performance rather than performance during a known evaluation.
- Track agreement over time and per annotator. A falling trend usually means fatigue or drift; a single divergent annotator usually means a correctable misunderstanding. Both are visible only if measurement is continuous.
Where that measurement becomes a contractual threshold — which metric, what number, sampled how, what happens when a batch fails — is covered separately in setting an annotation accuracy standard and SLA. The point here is narrower: without overlap and adjudication in the budget, there is no quality statement to put in an SLA at all.
What breaks when the modalities have to agree?
Single-modality work can be judged inside one signal. Multimodal work adds relationships: a spoken phrase aligning with a video event, a caption describing the correct region, a camera box and a LiDAR box referring to one physical object. The schema therefore needs cross-modal identifiers, timing rules, synchronisation tolerances, and a precedence rule for when signals disagree.
- Define a canonical timeline and state how each modality maps onto it, with the tolerance in frames or milliseconds written down.
- Carry shared IDs. Object identity has to be the same string in every stream, or the pairing cannot be checked automatically.
- Document coordinate systems and calibration for sensor fusion, and re-verify them per capture session rather than per project.
- Say what a paired annotation describes — the whole scene, a region, an event, or a moment. Ambiguity here produces pairs that are individually correct and jointly meaningless.
Errors propagate, which is why a single blended accuracy figure hides the cause. A bad transcript makes an image–text pair look mismatched; a calibration error shifts every 3D box even where annotators followed the guideline perfectly. Report label correctness, temporal alignment, spatial alignment, completeness and semantic consistency separately, and pilot the pipeline end to end before scaling — multimodal rework has to be undone across every file already processed.
Model-assisted labelling: real savings, real bias
Pre-labelling with a model and having annotators correct the output is now standard, and the throughput gain is genuine. So is the cost. Annotators presented with a plausible suggestion accept it more often than they would have produced it themselves, and the effect is strongest where you least want it — on ambiguous items, where a confident-looking box resolves the annotator's uncertainty in the model's favour.
The consequence is that a model-assisted dataset can encode the pre-labelling model's blind spots and then train a model that inherits them — with the agreement statistics healthy throughout, because annotators agree with each other about accepting the same suggestions.
- Keep unassisted control batches. The divergence between them and the assisted stream is the measurement of the anchoring effect.
- Suppress low-confidence suggestions. Where the model is unsure, showing nothing produces better labels than showing a guess.
- Audit the accepted-without-change rate. A very high rate is a warning sign, not an efficiency achievement.
- Never pre-label with the model being evaluated. It manufactures agreement between the dataset and the system it is meant to test.
What actually drives the cost?
Four factors, only one of which is volume.
- Ambiguity density. The share of items needing adjudication. Clean data is cheap at any volume; ambiguous data is expensive at any volume.
- Taxonomy size and depth. Deep hierarchies multiply the boundary decisions an annotator makes per item.
- Required agreement level. Higher targets mean more overlap, more adjudication and more senior review time — the lever buyers control most directly and understand least.
- Specialist knowledge. Medical, legal, industrial or language-specific tasks need qualified annotators, which changes the labour pool and the price.
Adjudication load = Items sent to a senior reviewer ÷ Items labelled
Track that ratio from the first batch. It is the earliest reliable signal of whether a project is priced correctly, and it moves long before any agreement statistic does. For scale rather than precision: Grand View Research estimates the global data collection and labelling market at USD 6.3 billion for 2026, growing at a 28.4% compound annual rate through 2030 — a research-firm estimate, not an audited total.
How Lifewood approaches this
Lifewood runs annotation programmes on this structure — versioned guidelines, overlap budgeted from the start, chance-corrected agreement measured continuously, adjudication by senior reviewers — with dual-layer human-in-the-loop QA held to a 95%+ accuracy threshold. For multimodal and multilingual work the constraint is who is available to judge: 50+ languages and 40+ delivery centres across 30+ countries mean audio and text are adjudicated by people who hear when something is subtly wrong, rather than by a reviewer working from a translated guideline. The AI-data heritage runs to 2004, with the current company established in 2018.
See global AI data, the QA process, autonomous driving annotation for the sensor-fusion case, and AI data validation.
Sources and further reading
- Landis, J.R. and Koch, G.G., "The Measurement of Observer Agreement for Categorical Data", Biometrics 33(1), 159–174, 1977 — the origin of the agreement bands quoted above.
- Data Collection and Labeling Market Size Report, 2025–2030, Grand View Research — a market-research estimate, not an audited figure.