How Do I Prepare My Data Before Sending It to an Annotation Vendor?

Buyer-side guides across the industry keep converging on the same diagnosis. iMerit's buyer's guide lists being vague on quality metrics and not including sample data as RFP mistakes that cost you later, because without a sample, vendors cannot estimate complexity or cost accurately. CloudPano's evaluation guide calls relying on a vendor's demo data instead of testing on your actual dataset the most common evaluation mistake. The pattern in both: the project is priced, staffed and scoped against data that is cleaner, simpler and more uniform than what eventually arrives.

Then production data lands. It has three date formats. Ten per cent of the audio is phone-quality, not studio-quality. The "English" dataset is a third code-switched. Fields the guideline assumed are empty half the time. None of this is unusual, but every discovery after kickoff costs a renegotiation, a guideline rewrite or a quality dispute. Discovered before kickoff, each one costs a sentence in the brief. Preparation is the work of moving those discoveries to the cheap side of the contract.

A task definition is a one-page statement of what the model will do with the labels, what the annotator sees, what they produce, and two or three examples of a wrong answer. The purpose matters: annotators who know a bounding box feeds a safety system draw it differently from one feeding a photo-tagging feature.

• A representative sample. Enough to price from and pilot on, typically 300 to 1,000 items for most tasks.
• Format and schema documentation. File types, encodings, field meanings and a stable unique identifier per item, so labelled results can be joined back to your systems without guesswork. If your data needs restructuring to be labelable at all, raise it explicitly; it is adjacent to the cleanup work in corpus cleaning and deduplication.
• Your privacy position in writing. What personal data the set contains, what you have scrubbed, what remains and on what basis, and any residency or handling constraints. This cannot be delegated.
• A draft guideline. Not a polished one, since a "final" guideline at handoff is a warning sign, but a genuine first draft with worked examples and counterexamples.
• A question channel and a decision-maker. The highest-leverage item on the list. Annotation surfaces ambiguity daily, and a named person who answers edge-case questions within a working day keeps a pilot moving. Sigma.ai's vendor-selection guidance makes the mirror-image point about vendors: good ones ask smart questions during the pilot, and a vendor that asks none is a red flag. A buyer who cannot answer questions is the most expensive kind.

A representative sample is a subset of your data that reproduces the variation of the full production pipeline, including its hard cases, rather than only its typical ones. The instinct is to send your best data. That sample produces an accurate quote and a successful pilot for a project that does not exist. The sample that protects you is stratified across everything that varies in production:

• Time. Pull from several periods, not last week. Formats, products and user behaviour drift.
• Source. Every channel, device type, region or business unit that feeds the real pipeline, roughly in proportion.
• Difficulty. Deliberately include the hard tail: blurry images, overlapping speakers, ambiguous tickets, the rare classes. If 5% of production is hard, make it 15 to 20% of the sample so the vendor prices and plans for it. Finding that tail systematically is its own discipline, covered in long-tail and edge-case mining.
• Language and locale. For multilingual work, sample per locale, not per language. A "Spanish" sample drawn only from Spain mis-prices a dataset that is 60% Latin American. For multilingual scoping, see managed multilingual data collection.

A representative sample makes the quote higher and the pilot slower, and both numbers are true, which is the point.

One more habit worth copying from the strongest buyers: label 30 to 50 items yourself first, internally, before sending anything. Nothing exposes a fuzzy task definition faster than your own team disagreeing about the answers. It also gives you the seed of a gold set, a gold set being a collection of items with known-correct labels that quality measurement is later scored against, as explained in Gold Sets, Audit Sampling and Consensus.

• Decide what personal data travels. Names, faces, voices, account numbers, and free-text fields where users type anything. For much annotation work, pseudonymising or redacting before handoff is both the safer and the simpler path, since annotators rarely need the real customer name to label intent. Where the task genuinely requires personal data, such as speaker verification or face-related work, consent and purpose documentation must cover annotation by a third party, and the handling constraints belong in the contract, not in an email.
• Match the data's sensitivity to the vendor's delivery model. Open crowd platforms and controlled delivery centres are different security propositions, unpacked in Secure Annotation of Sensitive Data: Controlled Centres vs Crowd Work. The preparation step is where you decide which your data requires, alongside any residency rules.
• Do the mechanical cleaning you can do cheaply. Deduplicate exact copies, remove corrupt and zero-byte files, standardise encodings, and verify that every item opens. None of this needs an annotation vendor, all of it otherwise gets billed back at annotation rates, and duplicates quietly distort both your quote and your eventual quality statistics.
• Leave the judgement cleaning alone. Do not pre-filter "confusing" items by eye. The confusing items are information about your data, your guideline and your task, and removing them produces the convenient sample described above.

An annotation guideline is the written specification that tells annotators how to apply each label, including definitions, examples and rules for ambiguous cases. A guideline written entirely before any annotator touches the data is a hypothesis. It will be wrong in ways only contact with real items can reveal, which is why writing annotation guidelines that annotators actually follow treats the first draft as exactly that, a hypothesis to be revised against the pilot. Arriving with a sealed "final" document signals that pilot feedback has nowhere to go.

What the draft must contain at handoff:

• The label set with definitions, and explicit boundaries between confusable labels, because boundaries are where annotators disagree.
• Worked examples and counterexamples, at least three per label, drawn from your real sample, with a sentence on why. Examples do more work than prose, so lead with them.
• An ambiguity rule: what the annotator does when genuinely unsure (a flag, an "unsure" label, an escalation), so uncertainty becomes data instead of silent guessing.
• The quality bar in numbers: the accuracy standard you will hold the work to, defined well enough to audit. "99% accuracy" with no denominator or method is not a standard, as argued in What Accuracy Standard to Require From an Annotation Vendor. Name the metric, the sampling method and the adjudication process.

The draft should not try to pre-answer every edge case from imagination. Leave room, and budget one or two guideline revision cycles into the pilot. The vendors worth working with will expect exactly that rhythm.

The intake covers the task definition on one page, a stratified sample reviewed jointly, a format-and-identifier check, a privacy and residency review against the client's constraints, and a pilot run under the draft guideline with gold-set items seeded from the client's own internally labelled examples. The question log the pilot produces becomes the first revision, and only then does volume pricing get fixed. Lifewood operates 40+ delivery centres across 30+ countries and covers 100+ languages through its AI data services, and the same intake runs per locale, because a sample that represents one market rarely represents them all.

This is not a house eccentricity; it matches the buyer-side guidance cited above. iMerit tells buyers to include sample data so proposals are comparable, CloudPano makes the pilot on your actual data the one evaluation step you cannot skip, and Sigma.ai frames vendor questions during the pilot as a quality signal. Whichever provider you choose, a vendor that quotes a firm per-unit price without seeing a representative sample is pricing someone's data, just not necessarily yours. For help comparing providers, see this ranking of the best human-in-the-loop AI companies for data annotation.

• Week 1: define and sample. Write the one-page task definition. Pull a stratified sample across time, source, difficulty and locale. Label 30 to 50 items internally and log every disagreement.
• Week 1: clean mechanically. Deduplicate, drop corrupt files, standardise encodings, attach stable IDs. Do not hand-filter hard cases.
• Week 2: settle the privacy position. Scrub or pseudonymise what annotation does not need, document the basis for what remains, and write down residency and handling constraints.
• Week 2: draft the guideline. Labels, boundaries, worked examples from your own sample, an ambiguity rule and a measurable quality bar.
• Before sending: set the channel. Name the person who answers annotator questions and commit to a response time.
• At handoff: share the disagreements too. The items your own team argued about are the most valuable items in the package. Send them flagged, not hidden.