Short answer. Nine criteria decide whether an AI data annotation partner will hold up at foundation-model scale: a defined quality measurement (gold sets and inter-annotator agreement, not a claimed accuracy percentage), modality and task fit, workforce model and annotator retention, throughput and ramp behaviour, multilingual and low-resource coverage, domain expertise for specialist tasks, security and data residency, tooling and integration, and governance — how taxonomy changes, edge cases and rework are handled. The first is the one most buyers under-specify and the one that predicts the most rework.
Annotation buying is unusually easy to get wrong, because the deliverable looks the same whether it is good or not. A labelled dataset arrives on schedule, passes a spot-check, and the cost of the errors inside it surfaces months later as a model that underperforms in exactly the segment where the labels were weakest.
These nine criteria are the ones that separate vendors after the demo. They are written to be used as an RFP structure — each has a specific artefact to request.
1. Quality measurement that is actually defined
"99% accuracy" is not a quality claim. It is a number with no denominator. Ask which of the following it refers to, because they measure different things and differ by a lot on the same dataset:
| Measure | What it tells you | When to require it |
|---|---|---|
| Gold-set accuracy | Agreement with a trusted reference set | Always — the baseline check |
| Inter-annotator agreement (IAA) | Whether two competent people labelling the same item agree | Any subjective or judgement-heavy task |
| F1 / precision / recall against gold | Detection quality, split by error type | Detection, segmentation, extraction |
| IoU thresholds | Geometric tightness of boxes and masks | Computer vision |
| Word error rate (WER) | Transcription quality | Speech |
| Kappa / Krippendorff's alpha | Agreement corrected for chance | Sentiment, ranking, moderation, RLHF |
The correction for chance matters more than it sounds. On a binary task with an unbalanced class distribution, raw agreement can look excellent while the chance-corrected figure shows the annotators are barely distinguishing anything:
Cohen's kappa = (Observed agreement − Expected agreement) ÷ (1 − Expected agreement)
F1 = 2 × (Precision × Recall) ÷ (Precision + Recall)
What to request: the vendor's quality definition per task type, the gold-set protocol (who builds it, how often it refreshes, what share of work is gold-injected), and last quarter's figures for a comparable project. A vendor without a gold-set protocol is inspecting output rather than measuring it.
Also ask what happens below threshold. A defined rework policy — who pays, at what turnaround, and how the root cause is fed back into training — separates a supplier from a subcontractor.
2. Modality and task fit
Annotation is not one skill. A vendor strong in 2D bounding boxes may be weak in LiDAR, and one strong in transcription may have no RLHF practice at all. Map your actual needs:
- Computer vision — 2D/3D bounding boxes, semantic and instance segmentation, keypoints, tracking across frames, LiDAR point-cloud and sensor-fusion labelling.
- Speech and audio — transcription, speaker diarisation, phonetic labelling, prosody, accent and dialect coverage.
- Text and NLP — entity recognition, intent classification, sentiment, summarisation quality.
- LLM and generative — RLHF preference ranking, SFT demonstration writing, red-teaming, response evaluation, data distillation.
- Content moderation — policy application at scale, with the wellbeing considerations that come with it.
Ask for a reference project in your modality, at your volume. Adjacent experience is a much weaker signal here than in most categories.
3. Workforce model and annotator retention
The workforce model determines quality stability more than the tooling does. Three broad models:
| Model | Strength | Weakness |
|---|---|---|
| Open crowd | Elastic, cheap, fast to start | High churn, weak on domain tasks, variable IAA |
| Managed workforce in owned centres | Trainable, retainable, auditable, secure | Slower to scale into a brand-new skill |
| Specialist contractors | Deep domain competence | Expensive, limited throughput |
Most enterprise programmes need the second, with the third layered in for specialist review. The question that reveals the truth: "what is your annotator retention rate on a project of our length, and what happens to quality when a project team turns over?" Every complex taxonomy has a learning curve; a vendor with high churn pays that curve repeatedly, and you pay for it in rework.
4. Throughput and ramp behaviour
Steady-state throughput is the easy number. The ones that matter:
- Ramp time to full quality at your volume — including the period where throughput exists but IAA has not stabilised.
- Peak behaviour. What happens when you triple volume for six weeks? Quality tracks reviewer load with a lag of roughly one cycle.
- Parallelism limits. How many distinct tasks can run at once without competing for the same trained pool?
Effective throughput = Items delivered × First-pass acceptance rate ÷ Cycle time
Ask for effective throughput, not delivered volume. A vendor delivering 100,000 items a week at 70% acceptance is a 70,000-item vendor charging for 100,000.
5. Multilingual and low-resource coverage
For foundation-model work this is frequently the binding constraint. Every vendor covers English, Mandarin, Spanish, French and German. Programmes are decided in Thai, Vietnamese, Bahasa Indonesia, Tagalog, Bengali, Swahili, the Arabic dialects and the long tail beyond.
Measure it as native-speaker annotator headcount per language, with location, not as a supported-language count. For speech work in particular, ask about dialect and accent coverage inside a language — a "Vietnamese" capability that is entirely Hanoi-based is not general Vietnamese coverage, and the resulting model will show it.
Low-resource languages carry a second requirement: the vendor needs a sourcing method, not just a roster. Ask how they recruit and validate speakers in a language they do not currently cover, and how long it takes.
6. Domain expertise for specialist tasks
Medical imaging, legal, financial, engineering and safety-critical driving scenarios need annotators who understand the content, not just the tool. Ask how domain reviewers are qualified, whether qualification is verified or self-declared, and what the escalation path is when an annotator is unsure.
The escalation path is the informative part. A programme with no defined route for "I don't know" produces confident wrong labels, which are more damaging than gaps because they are invisible in an acceptance check.
7. Security, privacy and data residency
Ask for evidence, not badges. Specifically:
- Where is data stored and processed, and can work be confined to a named jurisdiction or a specific facility?
- Which sub-processors touch it, and are they named?
- Physical controls where the data warrants them — secure rooms, no personal devices, no removable media.
- Access model — least privilege, revocation on rotation, audit logs.
- Certifications — ask for the certificate and the scope statement, not the logo. Scope is where these usually fall apart: a certification covering a corporate head office says nothing about the delivery centre doing your work.
- PII handling and the deletion path at project end, with confirmation.
8. Tooling and integration
Two viable models: the vendor's platform, or your platform operated by their workforce. Both are fine; ambiguity is not. Clarify who owns the annotation tool, whether your team gets read access to work in progress, and how the data actually moves — formats, schema versioning, API or bulk transfer, and how a mid-project taxonomy change propagates.
Ask whether the vendor can operate your tooling. A vendor who can only work inside their own platform creates a switching cost you are agreeing to at signature.
9. Governance: change, edge cases and disputes
The criterion that separates a two-year partner from a one-project supplier.
- Taxonomy change management. Real projects change definitions mid-flight. Ask how a change is versioned, whether previously labelled data is re-labelled or marked as a prior version, and who pays.
- Edge-case handling. Where do ambiguous items go, who adjudicates, and how do decisions become guideline updates rather than tribal knowledge?
- Guideline ownership. Written, versioned, and shared — or held in a project manager's head?
- Dispute resolution. When you disagree about whether a batch meets spec, what is the process before it becomes a commercial argument?
A scoring frame
| Criterion | Weight | Artefact to request |
|---|---|---|
| Quality measurement | 20% | Quality definition per task, gold-set protocol, last-quarter figures |
| Modality and task fit | 15% | Reference project in your modality at your volume |
| Workforce model and retention | 15% | Retention rate; quality behaviour across team turnover |
| Throughput and ramp | 10% | Ramp curve; peak behaviour; effective throughput |
| Multilingual coverage | 15% | Annotator headcount per language, with location |
| Domain expertise | 5% | Qualification method; escalation path |
| Security and residency | 10% | Certificates with scope statements; residency options |
| Tooling and integration | 5% | Format and schema handling; ability to run your tooling |
| Governance | 5% | Guideline versioning; taxonomy change and dispute process |
Run a paid pilot before committing volume — several thousand items including your hardest edge cases and at least one difficult language — and score the pilot on the same frame. A pilot reveals ramp behaviour and escalation quality, which no proposal can.
How Lifewood approaches this
Lifewood delivers annotation through a managed workforce in owned delivery centres rather than an open crowd, which is the model that suits long-running programmes with complex taxonomies — the learning curve is paid once and retained.
Scope spans the modalities above: LLM work including RLHF, SFT and response evaluation; computer vision including 2D/3D boxes, segmentation and keypoints for autonomous driving and medical imaging; speech and NLP including multilingual transcription and phonetic labelling with a specialism in low-resource languages and regional dialects; conversational-AI training data; content moderation; and bespoke field data collection.
The multilingual position is the structural one: 50+ languages, 40+ delivery centres across 30+ countries, and a global pool of 56,788 contributors, with region-native annotators rather than remote approximations. The AI-data heritage runs to 2004, with the current company established in 2018; engagements span frontier-model labs, voice-AI developers, AI compute vendors, computer-vision suppliers and autonomous-mobility programmes, with client identities withheld by agreement.
See AI data services, enterprise LLM training data, autonomous driving annotation, low-resource speech data, AI data validation and QA process.
Sources and further reading
- Cohen's kappa and Krippendorff's alpha are standard chance-corrected agreement measures; use them for any task involving judgement rather than raw agreement percentages.
- Lifewood service scope is published on lifewood.com/ai-services; delivery figures (50+ languages, 40+ centres, 30+ countries, 56,788 contributors) on lifewood.com.