Short answer. Yes, you can switch data annotation vendors without hurting your model, provided three assets live with you and not the outgoing vendor: the annotation guideline, the gold set, and the quality record. With those, a switch is a re-qualification exercise: the new vendor passes your gold set at your existing threshold, runs an overlap period, and only then takes volume.
Key takeaways
- Switching annotation vendors is safe when the buyer owns the guideline with its adjudication case law, the gold set, the quality record, and the data in open formats, and hazardous when those live only with the incumbent.
- Models get hurt by three mechanisms during a vendor change: silent guideline drift between workforces, an unmeasured quality step, and a coverage gap under deadline.
- The safe sequence is baseline and freeze, gold-set qualification at the incumbent's measured bar, a two-to-four-week overlap, a ramp by difficulty with doubled QA, and a documented close-out.
- A production switch typically takes six to twelve weeks and carries three costs: double payment during overlap, elevated QA, and a temporary throughput dip.
- Adjudicating overlap disagreements between the two vendors converts the old vendor's tacit knowledge into written guideline, which is the asset whose absence makes switching risky.
Why do teams switch annotation vendors, and why does it feel risky?
Teams switch for ordinary commercial reasons: quality that drifted, turnaround that slipped, a language or modality the current vendor cannot staff, a price that no longer survives procurement, or a security requirement the delivery model cannot meet. The switch feels risky because the accumulated understanding of your task often lives in the incumbent's reviewers rather than in a document.
Another common trigger is a consolidation decision of the kind covered in our annotation vendor consolidation and RFP guide. None of these reasons is exotic, and long-running data programmes outlive individual vendor relationships as a matter of course. If you are building a shortlist, the ranking of top AI data annotation companies is a reasonable starting point.
The fear has a real basis. CloudPano's vendor checklist singles out long-term programmes "where switching vendors mid-project is costly" as the case deserving extra diligence up front. The cost it points at is not the new contract but the undocumented understanding of your task that sits in the incumbent's team.
Undocumented task knowledge is the working interpretation of your guideline that annotators apply but that was never written down. It is the risk whether or not you ever switch: a task definition that exists only as vendor habit is a quality incident waiting for its first reviewer turnover, even at the same vendor. The assets that make switching safe also make staying safe, so building them is programme hygiene, not disloyalty.
What actually damages a model during a vendor change?
Three mechanisms damage a model, and none of them is the change of vendor name itself. They are silent guideline drift, an unmeasured quality step, and a coverage gap under deadline, and all three are preventable by design.
Silent guideline drift. The written guideline says one thing, but the incumbent's annotators, through months of adjudication decisions and chat answers, actually apply a refined version. A new vendor trained on the written document faithfully reproduces the old interpretation, and your dataset now holds two populations labelled under slightly different rules. The model sees contradictions it cannot resolve. The real guideline is the document plus its case law, and only the document got handed over; how to write annotation guidelines annotators actually follow covers the fix.
An unmeasured quality step. If you never held the old vendor to a defined accuracy standard (metric, denominator, audit method), you cannot tell whether the new vendor's first batches are better, worse or merely different. Our guide to the accuracy standard to require from an annotation vendor sets out that discipline. Teams in this position tend to discover quality changes months later in model metrics, which is the expensive place to find them.
A coverage gap under deadline. Switching with no overlap period, to hit a date, means the new vendor's learning curve lands directly in your training data. The hard tail of edge cases, rare classes and ambiguous items is where new teams err first and where models are most sensitive.
Notice what is absent: any inherent reason a competent second vendor cannot match a competent first one. Annotation quality is a function of guideline clarity, reviewer calibration and QA design, all of which are transferable, and not of vendor identity.
Which assets must you own before you can switch safely?
You need four assets: the guideline with its case law, a gold set, the quality record, and your data and labels in open formats. The time to secure them is while the current relationship is healthy.
The guideline, with its case law. This is the current document and the adjudication record: every edge-case ruling, every Q&A thread, every "we decided X counts as Y" decision, folded back into the written guideline as worked examples. If those rulings live in the vendor's ticketing system, request a consolidated export as routine documentation. Most contracts entitle you to it, and future contracts should say so explicitly.
The gold set. A client-owned, versioned set of items with adjudicated correct answers, large enough to qualify a workforce; several hundred items spanning the difficulty range is typical, built and maintained the way gold sets, audit sampling and consensus describes. It is the single most important portability asset, because it converts "is the new vendor as good?" from an opinion into a score.
The quality record. The incumbent's measured accuracy and inter-annotator agreement over time, on your definitions. Inter-annotator agreement is the rate at which independent annotators give the same label to the same item, and it is the half of the record that accuracy against a gold set cannot supply on its own. These numbers are the baseline the new vendor must match; without them, "no quality dip" is unfalsifiable. If you have never measured agreement, Cohen's kappa and Krippendorff's alpha are where to start, before the switch and not during it.
Your data and deliverables, in open formats. Raw items, labels and the mapping between them should be in formats you can hand to anyone, not locked inside a proprietary platform export you cannot read. Check this now; discovering a locked format during a dispute is the worst timing available.
The table below shows who holds each asset and what that means for the switch.
| Asset | If you own it | If the incumbent holds it |
|---|---|---|
| Guideline + adjudication case law | New vendor trains in days | Task definition must be reverse-engineered from data |
| Gold set with adjudicated answers | Qualification is a score | Qualification is an argument |
| Accuracy / agreement history | "Match the baseline" is enforceable | "No quality dip" is a hope |
| Data and labels in open formats | Transfer is a file copy | Transfer is a negotiation |
Every row is also a staying-put risk: an asset only the vendor holds is a single point of failure even if you never switch.
What does a safe transition plan look like, step by step?
A safe transition has five phases: baseline and freeze, qualification, overlap, a ramp by risk, and close-out. The shape mirrors how any new annotation workforce is stood up, with the old vendor's live output as an extra calibration reference.
Phase 1: Baseline and freeze. Pin the current guideline version. Run a fresh audit of the incumbent's recent output against the gold set so the baseline is current, not remembered. Freeze guideline changes for the duration of the transition: changing the task and the workforce simultaneously makes any quality movement uninterpretable.
Phase 2: Qualification. The new vendor's annotators train on the guideline and sit the gold set blind. The pass bar is the incumbent's measured baseline, not a round number and not the new vendor's own standard. Treat this like the paid pilot that buyer guides recommend for a first engagement; a switch is a pilot with a sharper benchmark.
Phase 3: Overlap. For two to four weeks, both vendors label an overlapping sample of live items. You get three measurements at once: new vendor against gold set, new vendor against old vendor, and, often the revelation, old vendor against gold set, since incumbents drift too. Disagreements between the vendors on the same items are the fastest map of where the written guideline and the lived guideline diverge. Adjudicate them, write the rulings into the guideline, and you have converted the incumbent's tacit knowledge into a document, which was the whole problem.
Phase 4: Ramp by risk. Shift volume to the new vendor starting with the item types where overlap agreement was highest, and hold the hardest classes with the incumbent longest. QA sampling runs at elevated rates (double is typical) through the ramp, tapering back to steady state only after agreement holds for consecutive weeks.
Phase 5: Close out. Take a final export of all data, labels and adjudication records, and add a dated note in the dataset documentation marking the transition window. If model metrics ever wobble, the first question, "did the data source change here?", is then answerable in seconds.
How long does a vendor switch take, and what does it cost?
For a running production programme, plan six to twelve weeks end to end. Simple, well-documented tasks compress toward the bottom of that range, while specialist domains and multi-language programmes extend toward the top.
The schedule breaks down as one to two weeks for baseline and freeze, two to three for qualification, two to four of overlap, and the remainder for ramp. Multi-language programmes run longer because qualification runs per language.
The cost has three honest components, and pretending otherwise is how switches get under-planned:
- Double payment during overlap. Two vendors label the same sample for a few weeks. This is the insurance premium; the claim it prevents, retraining a model on months of inconsistent data, costs more.
- Elevated QA. Your own review time roughly doubles during qualification and ramp.
- Throughput dip. Expect the new vendor at 60 to 80 percent of mature throughput in the first weeks, following the ramp-time-to-quality pattern in how to scale AI data annotation from pilot to production.
Against that, set the quiet costs of not switching when the reasons are real: paying a premium price for drifting quality, or building the next year's dataset on a vendor who cannot staff your next language. The comparison is rarely close once both sides are written down. For the price side, how to compare data annotation vendor quotes shows how to normalise offers.
How does Lifewood handle incoming annotation transitions?
Lifewood Data Technology treats an incoming transition as a re-qualification exercise, following the same sequence described above. Some annotation engagements begin as transitions rather than greenfield programmes, so the sequence is standing practice rather than a special arrangement.
In practice that means qualification against the client's own gold set and baseline, an overlap window against the outgoing vendor's live output where the client can arrange one, adjudication of every overlap disagreement into a revised guideline, and a ramp by item difficulty with elevated QA until agreement holds. Lifewood's published quality terms include a 95%+ accuracy SLA and two independent review passes with timestamped approval records. Where a client arrives without portable assets, such as a gold set or a measured baseline, the first deliverable is building them, because qualifying a new workforce against an undocumented standard protects nobody.
The same portability applies on the way out: guidelines, gold sets, adjudication records and labels are delivered in open formats as client property, which is what we would advise any buyer to require of any vendor, ourselves included. Multi-language transitions can draw on Lifewood's multilingual data collection work across 100+ languages from 40+ delivery centres in 30+ countries, and the wider AI services range covers the other modalities.
Industry buyer guidance points the same way. CloudPano's checklist flags switching cost as the reason to over-invest in initial evaluation, and Sigma.ai frames vendor accountability around measurable SLAs and KPIs. A provider on either side of a transition who resists an overlap period or a gold-set qualification is telling you something useful about how they expect to be measured.
What should you do if you are considering a switch right now?
Secure the portability assets first and decide second. Then re-baseline the incumbent, freeze the guideline, and insist on an overlap period before any volume moves.
- Secure the assets first. Export the guideline case law, verify the gold set is yours and current, pull the quality history, and confirm your data and labels are in open formats, all before giving notice.
- Re-baseline the incumbent. Audit their current output against the gold set this month. Your decision, and your pass bar for the new vendor, should rest on today's quality, not last year's.
- Freeze the guideline for the transition window. Change one variable at a time.
- Insist on overlap, and budget the double payment. Two to four weeks of parallel labelling is the cheapest model insurance available.
- Adjudicate overlap disagreements into the guideline. This is where the incumbent's tacit knowledge becomes your document.
- Ramp by difficulty, QA at double rate, and date-stamp the transition in your dataset documentation.
- Fix portability in the new contract. Name guideline, gold set, adjudication records and deliverable formats as client property, so the next switch, years from now, is this easy too.