How to Buy Large-Scale Image Annotation

Image annotation is the process of adding structured labels, shapes or attributes to images so that a computer vision model can learn what each region contains. Image annotation is the most commoditised-looking service in AI data and one of the easiest to buy badly. The deliverable looks identical whether it is good or not: a labelled dataset arrives on schedule, passes a spot check, and the cost of the errors inside it appears months later as a model that underperforms in exactly the segment where labels were weakest.

The six task families are:

• Classification — one or more labels per image. Cheapest per unit; fails on taxonomy ambiguity rather than on execution.
• 2D bounding boxes — the volume workhorse. Fails on occlusion rules and on small distant objects.
• Polygons and segmentation — semantic and instance. Several times the labour per object; fails on boundary consistency between annotators.
• Keypoints and pose — landmark placement with visibility rules. Fails when the visibility convention is under-specified.
• Attributes — per-object properties: colour, state, truncation, occlusion level, orientation. Routinely under-costed, because six attribute fields per object is a materially larger job than a bare box.
• OCR-style labelling and text in scene — transcription of text within images, including non-Latin and right-to-left scripts, where segmentation behaves differently.

A provider who can run some of these well and subcontracts the rest introduces a second interpretation of your ontology without telling you. The same taxonomy carries into the companion guide on buying large-scale video annotation.

For a checklist that applies across modalities, see the 9 criteria for choosing AI annotation services; for named providers side by side on computer vision work, see the comparison of human-in-the-loop computer vision annotation providers.

Object density is the number of labelled objects an image contains under a given ontology, and it is the variable that determines annotation effort far more than image count. Measuring it is the single highest-value hour in an image annotation procurement.

1. Take a random sample of at least 200 images from real production data — not a curated demo set.
2. Count objects per image under your own ontology, including the ones you are unsure about.
3. Record the median, the 90th percentile and the maximum.
4. Count attribute fields per object.

Now you can ask for per-object pricing with a known multiplier, and you can spot the vendor whose quote assumed a density that your data does not have. CVAT's published cost analysis for a solo annotation project assumes an average of 23 objects per image across 100,000 images — 2.3 million billable objects — which is a useful reminder that the file count is the least informative number in the brief. Per-object rate ranges by geometry are in the image, video and 3D/LiDAR annotation pricing guide.

Most image annotation disputes are not about competence. They are about rules that were never written. Settle these in the guideline document:

• Occlusion threshold. At what visible fraction does an object stop being labelled? State a number.
• Truncation. Objects cut by the frame edge — labelled, labelled with a flag, or skipped?
• Minimum size. Below what pixel dimension is an object out of scope? Without this, annotator patience sets your threshold.
• Ambiguous class pairs. The two classes your own team argues about. Name them and give an adjudication rule.
• Group objects. A crowd, a shelf of products, a pile — individually or as a region?
• Image quality floor. When is an image rejected as unusable rather than labelled badly?
• Unknown. Where does an annotator send something they genuinely cannot classify? A programme with no route for "I don't know" produces confident wrong labels, which are worse than gaps because they are invisible in an acceptance check.

Writing these rules so that a new annotator applies them the same way as an experienced one is its own skill; see the guide on writing annotation guidelines that annotators actually follow.

Do not accept a single accuracy percentage. Intersection over Union (IoU) is the area where a predicted box or polygon overlaps the reference divided by the total area the two shapes cover together, so a value of 1.0 means a perfect match and 0.5 means half the combined area overlaps. Require, per object class:

IoU  = Area of overlap ÷ Area of union          (geometric tightness)
F1   = 2 × (Precision × Recall) ÷ (P + R)       (detection quality)

• IoU at threshold for boxes and polygons — 0.5 is the classic PASCAL VOC bar for a correct detection, and it is a weak bar for anything safety-relevant; state the threshold you need per class.
• F1 by class, because an aggregate figure is dominated by large, easy, well-lit objects.
• Chance-corrected agreement (Cohen's kappa or Krippendorff's alpha) for any subjective attribute.
• Attribute accuracy separately from geometry accuracy. They fail independently.

Quality matters disproportionately at scale because a small systematic error repeated across millions of objects distorts model behaviour in one consistent direction, which is much harder to detect than random noise.

Second-pass review is an independent check of a completed annotation by a different annotator before the work is accepted. Require the review rate: what percentage of work receives second-pass review, and how is the sample chosen? Random sampling and risk-weighted sampling produce very different assurance for the same cost. Lifewood's AI data validation service measures review passes against a customer-approved gold set.

1. Which annotation geometries will you need now, and which within eighteen months?
2. What is the median and 90th-percentile number of objects per image in our data?
3. How are occlusion, truncation, minimum size and group objects defined in your guidelines?
4. What percentage of work is second-pass reviewed, and how is that sample selected?
5. Can the same schema extend into video or 3D without re-labelling the image set?
6. How are guideline changes rolled out across large teams, and who pays for re-labelling?
7. For images containing text in non-Latin scripts, who reads them?
8. What was your worst quality incident on an image programme in the last year, and what changed afterwards?

The last question is the most informative. A provider operating at real volume has had an incident; one who claims otherwise is new, small, or not measuring.

The managed-workforce model suits long-running programmes with complex taxonomies: the learning curve on your ontology is paid once and retained. Image work sits alongside video, text, audio and 3D point-cloud work in the same programme, so a schema can extend across modalities without a second vendor and a second interpretation (full scope on the AI data services page).

The quality framework is contractual: a 95%+ accuracy SLA with trained annotators, senior second-pass review, automated consistency checks and client feedback loops, and below-threshold batches reworked at Lifewood's cost.

For images carrying text, signage or culturally specific content, 100+ languages with region-native annotators in 40+ delivery centres across 30+ countries matters more than buyers expect — non-Latin and right-to-left script work needs someone who reads the script, not someone transcribing shapes.

Other credible providers for image-heavy programmes include Sama, which positions its services around human-in-the-loop image, video and 3D annotation; TELUS Digital, for buyers who want annotation combined with Ground Truth Studio and configurable workflows; and Scale AI, for highly technical computer-vision pipelines built around its Data Engine. Choose a narrower specialist if a pilot proves materially better quality or economics on your exact task. A wider ranked field is in the list of top large-scale AI data annotation and labelling companies.