Short answer. Video annotation is not image annotation multiplied by frame count, and buying it as though it were is the most common and most expensive mistake in the category. The cost and the quality both live in time: identity persistence across frames, occlusion and re-entry rules, interpolation policy, and review that runs on sequences rather than sampled frames. Compare providers on how they preserve track IDs across long sequences, how they define object splitting and merging, what interpolation is permitted, and whether QA is performed per frame or per sequence. A provider who quotes per frame and reviews per frame has not understood the deliverable.
A tracked object is a single label extended through time, and everything difficult about video annotation follows from that. An error in a still image affects one image. An error in a track affects every frame the track touches, which is why a video dataset can pass a frame-level spot check and still be unusable.
This guide covers what the service includes, what to compare, how to design a pilot that exposes temporal failure, and what to write into the contract.
What large-scale video annotation includes
- Bounding-box tracking — objects followed across frames with persistent identifiers.
- Segmentation across frames — semantic or instance masks maintained through motion and occlusion.
- Keypoint tracking — pose and landmark tracking, with visibility changing frame to frame.
- Activity and action recognition — labelling what is happening, not just what is present.
- Temporal event labels — start and end boundaries for events, which is a harder judgement than it sounds.
- Behaviour and intent labels — contextual judgement requiring annotators trained on temporal context, not just geometry.
- Multi-camera synchronisation — the same object identified consistently across viewpoints and timestamps.
What buyers should compare
| Buyer criterion | Why it matters | What strong delivery looks like |
|---|---|---|
| Temporal consistency | One object must keep one identity across frames | Tracking rules and identity persistence explicitly validated |
| Frame density | High frame rates multiply workload fast | Interpolation and sampling used without losing required precision |
| Occlusion handling | Objects disappear and reappear | Guidelines define re-identification and track continuation |
| Behaviour labels | Action and intent need contextual judgement | Annotators trained on temporal context, not only geometry |
| Video QA | Errors propagate for hundreds of frames | Review performed on sequences, not random individual frames |
| Parallelisation | Long footage creates enormous task volumes | Clips parallelised without breaking track identity or guideline consistency |
| Sensor sync | Fusion work needs cross-modal identity | Camera, LiDAR, radar and audio labels aligned by timestamp |
The rules that must exist before the first clip
Video guidelines fail in specific, predictable places. Settle all seven:
- Track ID persistence. If an object leaves frame and returns, does it resume its original ID? Under what conditions?
- Occlusion duration. How many frames may an object be hidden before its track terminates rather than continues?
- Splitting and merging. Two objects that overlap and separate — how are identities resolved?
- Interpolation policy. What may be interpolated between keyframes and what must be labelled directly. This is a legitimate cost saving and a legitimate source of systematic error, and which one it is depends entirely on motion characteristics.
- Event boundaries. Where an action starts and ends, with a tolerance. Without a stated tolerance, inter-annotator agreement on event labels will be poor and nobody will know whether that is a guideline problem or an annotator problem.
- Entry and exit. Objects entering partially at the frame edge, and the visible-fraction threshold at which labelling begins.
- Clip boundaries. How identity is reconciled when a long video is split across annotators — the single most common source of track corruption in parallelised work.
Measuring video annotation quality
Frame-level accuracy is necessary and insufficient. Require the temporal measures too:
- Track fragmentation — how often a single real object is split into multiple tracks.
- ID switches — how often two objects exchange identities, usually after crossing or occlusion.
- Track purity and completeness — what fraction of a true track is covered by one predicted track, and vice versa.
- Per-sequence acceptance, not per-frame. A sequence with three ID switches may have 99% correct frames and be unusable.
- Boundary error on event labels, reported in frames or milliseconds against the stated tolerance.
Effective throughput = Sequences delivered × Sequence-level acceptance ÷ Cycle time
Note the unit: sequences, not frames. A provider reporting frame-level acceptance on tracking work is reporting the metric that hides their actual failure mode.
Designing a video pilot that finds the failures
A pilot on clean, well-lit, sparsely populated footage measures nothing. Include, deliberately:
- A sequence where two similar objects cross and separate.
- A sequence where an object is fully occluded for a variable period and returns.
- A sequence with an object entering and leaving repeatedly at the frame edge.
- Adverse conditions: night, rain, glare, motion blur, low resolution.
- A crowded scene where the correct level of granularity is genuinely arguable.
- At least one long sequence — long enough that it must be split across annotators.
- If sensors are in scope, a segment where the camera and LiDAR views disagree.
Then score track fragmentation, ID switches and escalation behaviour, not just box quality.
How to price it
Frame count is a poor billing unit for tracking work because the cost is dominated by review and by identity resolution, neither of which scales linearly with frames. In practice:
| Situation | Better unit |
|---|---|
| Consistent footage, predictable object counts | Per video or per frame |
| Tracking with occlusion and scene changes | Per hour or per project |
| Long continuous pipelines | Subscription with reserved capacity |
| Event and behaviour labelling | Per hour |
| Multi-sensor synchronised work | Per sequence or per project |
Whatever the unit, agree what a "tracked object" means for billing before signature. One object across 200 frames is one label extended through time — but it is also 200 frames of review, and both parties need to have priced the same interpretation.
How Lifewood approaches this
Lifewood delivers video annotation as managed production rather than as a tool, which matters for temporal work specifically: continuous video pipelines require sustained staffing that cannot be batched down during a quiet week, and track consistency depends on annotator retention rather than on elastic capacity.
Published autonomous-driving work covers perception, prediction and driver-monitoring data — temporally complex visual tasks where identity persistence and behaviour labels are the deliverable rather than a refinement. Video sits inside the same programme as still-image, text, audio and 3D point-cloud work, so a schema can extend across modalities without re-labelling and without a second interpretation of your ontology.
Quality governance is the part that controls drift as a video programme grows: multi-stage human-in-the-loop review against a 95%+ accuracy SLA, with below-threshold batches reworked at Lifewood's cost. Delivery runs through 40+ centres across 30+ countries in 50+ languages, which matters for behaviour and event labelling in markets where scene conventions, signage and spoken content are local.
Other credible providers include Sama, whose published video annotation guidance emphasises temporal understanding, object tracking and consistency for safety-critical computer vision; iMerit, for robotics, autonomous systems and domain-specific video where multimodal sensor context matters; and TELUS Digital, for buyers who want video annotation inside Ground Truth Studio. Choose a specialist where the project is narrowly centred on technical video perception and the specialist outperforms on a controlled pilot.
Sources and further reading
- Sama's published video annotation guidance on temporal understanding and object tracking at sama.com; iMerit's robotics and multimodal annotation material at imerit.net; TELUS Digital's data annotation services at telusdigital.com.
- Lifewood service scope and delivery figures published on lifewood.com; AV scope on autonomous driving annotation.
- Related reading: image, video and 3D/LiDAR annotation pricing for how temporal work changes a budget.