Skip to main content
← All articles
AI Data

Model-Assisted Labelling and Active Learning: When Pre-Labels Help and When They Bias

Short answer. These are two separate decisions that get bundled together. Active learning decides which unlabelled items to send for annotation; pre-labelling decides what the annotator…

Mumu D. · August 2026 · 8 min read

Download PDF

Short answer. These are two separate decisions that get bundled together. Active learning decides which unlabelled items to send for annotation; pre-labelling decides what the annotator sees when the item arrives. The evidence on pre-labelling is genuinely positive — a clinical NER study measured time savings of 13.85–21.5% per entity with no statistically significant difference in agreement or annotator performance — but with one crucial caveat: errors surviving a pre-annotation workflow are systematic where human-only errors are random, and agreement scores can overstate quality precisely because every annotator saw the same suggestion.

Why separate selection from pre-labelling?

Because they fail in opposite directions. One skews which examples exist in your dataset; the other skews what the labels on those examples say.

Active learning rests on a simple principle: models can reach high accuracy with fewer labelled samples if you strategically select the most informative points to train on. Rather than annotating random documents to build a comprehensive corpus, the learner nominates the instances expected to benefit it most — commonly through uncertainty sampling, which measures learner confidence on unlabelled instances and queries the least confident.

Pre-labelling is a different intervention entirely. A trained model runs inference first, and the annotator receives a populated screen to accept, correct or reject. The two combine well, but they need to be evaluated separately: an active-learning strategy that over-samples one region of the feature space produces a training set that is not representative of the underlying distribution, while a pre-label that is subtly wrong produces a label the annotator never independently considered. Neither problem shows up in the other's metrics.

When do pre-labels genuinely help?

On high-volume, well-specified tasks where a competent model already exists — and the measured gains are real.

The strongest evidence comes from a JAMIA study that built a gold standard from 1,400 randomly selected clinical trial announcements, double-annotated for diagnoses, signs, symptoms and clinical codes, with pre-annotation drawn from dictionary-based methods and tested using F-measures, ANOVA and Bonferroni correction.

A second study reached a compatible conclusion on a harder task. Analysing dependency syntax annotation — mid-level complexity, pre-annotated with a high-accuracy parser — researchers found pre-annotation to be an efficient tool for faster manual annotation that increased the consistency of the resulting annotation without reducing its quality. Earlier Penn Treebank work pointed the same way, with the semi-automatic approach delivering both a significant reduction in annotation time and increased inter-annotator agreement and accuracy.

So the efficiency case is well supported. The nuance is in what "quality" was measured with.

When do they bias the data?

When the model is confidently wrong in a consistent way, and when you use agreement alone to check.

PRE-LABELS HELP WHEN PRE-LABELS BIAS WHEN

  • The task is well specified and the label set stable

  • The model is systematically wrong on a subgroup

  • A competent model or dictionary already exists

  • The task is subjective — sentiment, toxicity, preference

  • The work is high-volume and repetitive

  • Omissions matter: a missing span is invisible

  • Errors are visible — a wrong span is obvious on screen

  • Every annotator sees the same suggestion

  • Annotators are trained to reject, not just accept

  • Quality is judged by agreement alone The model does the typing.

Pre-annotation errors are systematic; human-only errors are random. Systematic errors survive averaging.

13.85–21.5% time saved per entity, with agreement between 93.4% and 95.5%.

Two findings deserve to be read together. First, the measurement problem: studies inferring high quality from pre-annotation often measure it with inter-rater agreement, which may overestimate quality when multiple annotators are influenced by the same pre-annotations. Two annotators agreeing because they both accepted the same model suggestion is not independent corroboration. Second, the error-shape problem: errors arising from pre-annotation workflows follow a more systematic pattern, whereas errors from human-only annotation tend to be more random — and random error washes out in aggregate while systematic error propagates straight into the model.

The cognitive mechanism is documented too. Human-in-the-loop review is not a neutral filter; it introduces additional biases because annotators carry cognitive biases of their own. Research simulating behavioural biases — including anchoring — within an active-learning loop on a real pancreatic cancer dataset found classification performance deteriorated significantly when human decisions were influenced by those biases, relative to an unbiased reference case. Anchoring is precisely what a populated screen invites.

How should you choose what to annotate next?

Deliberately, and with a check that your selected set still resembles the world.

Selection strategies, and what each one costs you STRATEGY HOW IT SELECTS THE RISK IT CARRIES VERDICT RANDOM SAMPLING Uniformly from the pool Inefficient — spends budget on examples the model already handles Your baseline and your control set UNCERTAINTY SAMPLING Lowest model confidence; points near the decision boundary Over- or under-samples regions, producing an unrepresentative training set Efficient, but watch coverage DIVERSITY / GRADIENT-BASED Batch selection balancing informativeness with spread across the space More computation; still no guarantee of subgroup coverage Better for batch labelling HYBRID + SEMISUPERVISED Active queries plus unlabelled data folded back into learning Added pipeline complexity Demonstrated to reduce sampling-bias effects Evidence is not one-sided: an empirical study found active set selection using posterior entropy from deep models robust to sampling biases and to query size and strategy choices, contrary to earlier literature. Test on your own data rather than assuming either result.

The compounding risk is worth naming plainly. Active acquisition assumes the labels it collects are sound, but many active-learning applications rely on human-generated labels that are highly bias-prone — and research into active data acquisition under label bias documents several patterns in which more data leads models astray rather than improving them. Volume does not correct a biased selection rule; it entrenches it.

One more caution specific to multilingual programmes: a model used for pre-labelling almost always performs worse in lower-resource languages, so the same workflow that saves 20% of the time in English can quietly anchor annotators to poor suggestions elsewhere. Per-language model evaluation before enabling pre-labels, and native-speaker review of the accepted labels, are exactly the human-in-the-loop discipline Lifewood applies across 50+ languages and dialects.

Evaluate the pre-label model before you trust it. Measure its error profile by class and subgroup — a model that is 92% accurate overall can be 40% accurate on the cases that matter.

Hold out a blind control set. Have a portion annotated from scratch, without suggestions, and compare. It is the only way to detect anchoring.

Do not judge pre-labelled data by agreement alone. Shared suggestions inflate agreement; pair it with accuracy against an independently annotated reference.

Track the accept rate as a warning signal. An acceptance rate near 100% means annotators have stopped reviewing.

Keep random sampling in the mix. A random slice alongside the active queries preserves a representative view and gives you an unbiased evaluation set.

Audit coverage, not just accuracy. Check that the selected set still spans your subgroups, languages and rare classes after each acquisition round.

Turn pre-labels off for subjective tasks. Sentiment, toxicity and preference judgements are where anchoring does the most damage and shows the least.

Evaluate the pre-label model per language. Enable suggestions only where the model has earned them.

Key takeaways

  • Active learning selects which data to annotate; pre-labelling shapes what the annotator sees. Evaluate them separately.
  • Pre-annotation saved 13.85–21.5% of annotation time per entity in a 1,400-document clinical NER study, with IAA of 93.4– 95.5% and no statistically significant difference in agreement or performance.
  • On dependency syntax annotation, pre-annotation increased consistency without reducing quality; Penn Treebank work found similar gains in time, agreement and accuracy.
  • The caveat: agreement may overestimate quality when every annotator sees the same suggestion.
  • Pre-annotation errors are systematic while human-only errors are random — and systematic error survives aggregation.
  • Simulated anchoring and related biases in an active-learning loop significantly degraded model accuracy versus an unbiased reference.
  • Uncertainty sampling can over- or under-sample regions, producing unrepresentative training sets; semi-supervised methods have been shown to reduce this.
  • Evidence is mixed: posterior-entropy selection with deep models proved robust to sampling bias in one large empirical study, so test on your own data.

Sources and further reading

Frequently asked questions

The direct measurements say no — clinical NER and dependency syntax studies both found no quality loss, and consistency improved in the latter. The open question is whether agreement-based metrics fully capture quality when annotators share the same suggestions.

Annotate a held-out slice from scratch with no suggestions, then compare against the pre-labelled output on the same items.

No. Sampling bias is a known issue in active-learning paradigms, but one large empirical study found posterior-entropy selection robust to it across query sizes and strategies. Treat it as a property to measure rather than assume.

You can, but the errors correlate: the model chooses items it finds uncertain and then anchors the annotator with its own guess on exactly those items. Keep a random slice and a blind control set as counterweights.

Have an AI or visibility project in mind?

From AI evaluation and human-in-the-loop review to GEO and AEO strategy, our team can help you deploy with confidence and get found in the AI search era.

Talk to our team