Short answer. Autonomous driving annotation is judged on the cases that almost never occur. A vendor that labels ordinary daylight highway frames to 99% accuracy and mishandles occluded pedestrians at dusk has delivered a dataset that trains a model to fail exactly where failure matters. Require six things: task-specific quality thresholds (IoU for cuboids, identity persistence for tracking, not a blended accuracy figure), explicit edge-case and ODD coverage design, sensor-fusion consistency across camera, LiDAR and radar, temporal consistency across sequences, a trained retained workforce rather than an open crowd, and auditable provenance with data residency control. Price per object is the least informative number in the negotiation.
Perception data is the most demanding annotation category in commercial use. The geometry is three-dimensional, the labels must stay consistent across time, several sensors must agree, and the consequences of systematic error are safety consequences rather than quality ones.
This guide sets out what an enterprise buyer should specify and verify.
What the work actually consists of
| Task | Output | Where it goes wrong |
|---|---|---|
| 2D bounding boxes | Object class and image-space box | Tight-fit inconsistency; truncation at frame edges |
| 3D cuboids on point cloud | Position, dimensions, heading in 3D | Heading errors on distant or sparse objects |
| Semantic segmentation | Per-pixel class | Boundary precision; ambiguous surface classes |
| Instance segmentation | Per-object masks | Overlapping and occluded instances |
| Lane and road structure | Polylines, topology, attributes | Faded markings; construction; merges and splits |
| Traffic signs and signals | Class, state, relevance | Which signal governs which lane; state during transition |
| Tracking across frames | Persistent identities | Identity switches through occlusion |
| Sensor fusion | Consistent labels across camera, LiDAR, radar | Calibration drift; disagreement between modalities |
| Behaviour and intent | Actor state, predicted action | Genuinely subjective; needs the strictest guidelines |
Each has a different failure mode, which is why a single accuracy percentage across a delivery tells you almost nothing.
1. Task-specific quality thresholds
Specify the metric per task, and the threshold, before work starts.
IoU = Area of overlap ÷ Area of union
F1 = 2 × (Precision × Recall) ÷ (Precision + Recall)
- Cuboids and boxes — IoU thresholds, stated separately for near and far range. Distant objects are sparse in the point cloud and a single global threshold either over-penalises far range or under-measures near range.
- Segmentation — mean IoU per class, not overall. Rare classes are where the value is; an overall figure is dominated by road and sky.
- Tracking — identity persistence through occlusion, and identity-switch count per sequence. This is not measurable frame by frame, which is why vendors who audit per-frame miss it entirely.
- Classification and attributes — F1 per class, plus a confusion matrix. Which classes get confused with which is more actionable than the headline number.
- Subjective tasks — chance-corrected agreement between independent annotators, because on intent and behaviour there is no gold answer, only a defensible consensus.
Ask what the vendor does below threshold: who pays for rework, at what turnaround, and how the root cause feeds back into annotator training rather than being fixed silently batch by batch.
2. Edge-case and operational design domain coverage
The most important part of the specification and the part most often left implicit. Define the ODD and require the dataset to be stratified against it, not merely sampled from whatever was recorded:
- Lighting — daylight, dusk and dawn, night, low sun directly into the sensor, tunnel entry and exit.
- Weather — rain, spray, fog, snow, wet reflective surfaces.
- Density — empty road through dense urban traffic.
- Actors — pedestrians including children, cyclists, motorcycles, wheelchairs, animals, unusual vehicles, people carrying or pushing objects.
- Occlusion — partial, heavy, and re-appearance after full occlusion.
- Road structure — construction zones, temporary markings, unmarked roads, complex junctions, roundabouts.
- Regional variation — signage, markings, vehicle types and driving conventions differ by country, and a model trained in one region degrades in another.
Stratum coverage = Annotated sequences in stratum ÷ Target sequences in stratum
Report the minimum across strata, not the mean. The mean says the programme is on schedule; the minimum names the condition in which the model will fail.
3. Sensor-fusion consistency
Where several modalities describe the same scene, the labels must agree. Three checks worth writing into the specification:
- Cross-modal identity — the same object carries the same identity in camera and LiDAR.
- Projection consistency — a 3D cuboid projected into the image plane lands on the object. Cheap to check automatically and a reliable detector of calibration drift.
- Disagreement handling — a defined rule for what happens when modalities conflict, rather than an annotator's ad hoc choice. Conflicts are informative and should be logged, not silently resolved.
4. Temporal consistency
Sequence data has failure modes that no per-frame audit finds: identities that switch through occlusion, dimensions that pulse frame to frame, headings that flip 180 degrees on symmetric objects, and interpolated frames that drift from reality between keyframes.
Require sequence-level review, and ask specifically how interpolation is used and validated. Interpolation between keyframes is a legitimate efficiency and a common source of quiet error.
5. Workforce model and retention
Perception taxonomies are complex and take weeks to learn. An open crowd pays that learning curve repeatedly; a retained, trained workforce pays it once.
Ask for annotator retention on projects of comparable length, and what happens to quality when a team turns over. Then ask about specialist review for safety-critical categories — who adjudicates ambiguous cases, and whether "I am not sure" has a defined escalation path. A programme without one produces confident wrong labels, which are more dangerous than gaps because they pass an acceptance check.
6. Provenance, residency and security
Driving data is recorded in public space and routinely contains faces and licence plates.
- Residency — can work be confined to a named jurisdiction or facility? Recorded-in-public data frequently cannot cross certain borders.
- Privacy processing — face and plate blurring, at what stage, and whether the original is retained and under what control.
- Access model — least privilege, revocation, audit logs; physical controls where warranted, including secure rooms and no removable media.
- Per-item provenance — who labelled it, who reviewed it, under which guideline version.
- Certifications — ask for the certificate and its scope statement, not the logo. Scope is where these usually fail: a certification covering a head office says nothing about the delivery centre doing your work.
How to run the pilot
A proposal cannot demonstrate any of the above. A paid pilot can, in about three weeks.
Build the pilot set deliberately: a majority of ordinary sequences, plus a deliberate minority of hard ones — night rain, heavy occlusion, a construction zone, an unusual actor, a sequence with a long full occlusion and re-appearance. Include at least one sequence you have already annotated internally to a standard you trust, and do not tell the vendor which one it is.
Score on: per-task metrics against your thresholds; identity switches across the occlusion sequence; projection consistency between modalities; per-class F1 on rare classes; how ambiguous cases were escalated and documented; and the guideline questions the vendor raised — a vendor asking sharp questions in week one is reading the taxonomy properly.
How Lifewood approaches this
Lifewood delivers perception annotation — 2D and 3D bounding boxes, semantic segmentation and keypoint labelling for autonomous driving and medical imaging — through a managed workforce in owned delivery centres rather than an open crowd, which is the model that suits complex taxonomies where the learning curve is the cost.
Two structural points matter for this category specifically. Owned centres across 40+ locations in 30+ countries make it practical to confine processing to a named jurisdiction, which recorded-in-public driving data often requires. And regional coverage across Asia, Europe, North America and Africa means datasets can be annotated by people who recognise local signage, markings and driving conventions rather than inferring them. Automotive and vision engagements span AI compute vendors, autonomous-mobility developers and computer-vision suppliers.
See autonomous driving annotation, AI data services, AI data validation, QA process and edge intelligence.
Sources and further reading
- Companion guide: 9 Criteria for Choosing AI Annotation Services — the cross-modality vendor evaluation frame this specialises.
- Lifewood perception scope is published at lifewood.com/autonomous-driving-annotation.