Short answer. These are two separate decisions that get bundled together. Active learning decides which unlabelled items to send for annotation; pre-labelling decides what the annotator sees when the item arrives. The evidence on pre-labelling is genuinely positive — a clinical NER study measured time savings of 13.85–21.5% per entity with no statistically significant difference in agreement or annotator performance — but with one crucial caveat: errors surviving a pre-annotation workflow are systematic where human-only errors are random, and agreement scores can overstate quality precisely because every annotator saw the same suggestion.
Key takeaways
- Active learning selects which data to annotate; pre-labelling shapes what the annotator sees. They fail in different ways and should be evaluated separately.
- Pre-annotation saved 13.85–21.5% of annotation time per entity in a 1,400-document clinical NER study, with inter-annotator agreement of 93.4–95.5% and no statistically significant difference in agreement or performance.
- On dependency syntax annotation, pre-annotation increased consistency without reducing quality; earlier Penn Treebank work found similar gains in time, agreement and accuracy.
- Agreement scores can overstate quality when every annotator sees the same model suggestion, because two people accepting one suggestion is not independent corroboration.
- Pre-annotation errors are systematic while human-only errors are random, so pre-annotation errors survive averaging instead of washing out.
- A simulation of anchoring and related cognitive biases inside an active-learning loop significantly degraded model accuracy compared with an unbiased reference case.
Why separate selection from pre-labelling?
Because they fail in opposite directions: one skews which examples exist in the dataset, the other skews what the labels on those examples say.
Active learning is a training strategy that lets a model reach high accuracy with fewer labelled examples by nominating the unlabelled items it expects to benefit from most, commonly through uncertainty sampling — ranking items by how unconfident the current model is and sending the least confident ones for labelling. Pre-labelling (also called pre-annotation) is a separate step in which a trained model runs inference first, and the annotator receives a populated screen to accept, correct or reject rather than a blank one. The two combine well in most human-in-the-loop pipelines, but an active-learning strategy that over-samples one region of the feature space produces a training set that no longer represents the underlying distribution, while a pre-label that is subtly wrong produces a label the annotator never independently considered. Neither failure shows up in the other's metrics, which is why teams building a repeatable human-in-the-loop machine learning process test the two stages against different checks.
When do pre-labels genuinely help?
On high-volume, well-specified tasks where a competent model already exists, and the measured gains are real rather than assumed.
The strongest evidence comes from a JAMIA study that built a gold standard from 1,400 randomly selected clinical trial announcements, double-annotated for diagnoses, signs, symptoms and clinical codes, with pre-annotation drawn from dictionary-based methods and tested using F-measures, ANOVA and Bonferroni correction. It found pre-annotation cut annotation time by 13.85–21.5% per entity with inter-annotator agreement — the measure of how consistently two independent annotators label the same item — holding between 93.4% and 95.5%, with no statistically significant bias effect.
A second study reached a compatible conclusion on a harder task. Analysing dependency syntax annotation — mid-level complexity, pre-annotated with a high-accuracy parser — researchers found pre-annotation to be an efficient tool for faster manual annotation that increased the consistency of the resulting annotation without reducing its quality. Earlier Penn Treebank work pointed the same way, with the semi-automatic approach delivering a significant reduction in annotation time alongside increased agreement and accuracy. Programmes that track inter-annotator agreement as a quality signal should read these results alongside the caveat below: the efficiency case is well supported, but the nuance is in what "quality" was actually measured with.
When do they bias the data?
When the underlying model is confidently wrong in a consistent way, and when agreement between annotators is the only check being used.
| Pre-labels help when… | Pre-labels bias when… |
|---|---|
| The task is well specified and the label set is stable | The model is systematically wrong on a subgroup |
| A competent model or dictionary already exists | The task is subjective — sentiment, toxicity, preference |
| The work is high-volume and repetitive | A missing span or omission is invisible on a populated screen |
| Errors are visible — a wrong span is obvious | Every annotator sees the same suggestion |
| Annotators are trained to reject, not just accept | Quality is judged by agreement alone |
Two findings deserve to be read together. First, the measurement problem: studies inferring high quality from pre-annotation often measure it with inter-rater agreement, which may overestimate quality when multiple annotators are influenced by the same pre-annotations — two annotators agreeing because they both accepted the same model suggestion is not independent corroboration. Second, the error-shape problem: errors arising from pre-annotation workflows follow a more systematic pattern, whereas errors from human-only annotation tend to be more random, and random error washes out in aggregate while systematic error propagates straight into the trained model. This is the same distinction that gold sets, audit sampling and consensus checks are built to catch, because none of them rely on agreement alone.
The cognitive mechanism behind this is documented too. Human-in-the-loop review is not a neutral filter; annotators carry cognitive biases of their own, and a populated screen invites anchoring — the tendency to lean on the first suggestion seen rather than reasoning independently. A simulation of anchoring, gambler's fallacy and regret-aversion inside an active-learning loop on a real pancreatic cancer dataset found classification performance deteriorated significantly relative to an unbiased reference case.
How should you choose what to annotate next?
Deliberately, with a selection strategy matched to the task, and with a standing check that the selected set still resembles the world it is meant to represent.
| Strategy | How it selects | The risk it carries | Verdict |
|---|---|---|---|
| Random sampling | Uniformly from the pool | Inefficient — spends budget on examples the model already handles | Your baseline and control set |
| Uncertainty sampling | Lowest model confidence; points near the decision boundary | Over- or under-samples regions, producing an unrepresentative training set | Efficient, but watch coverage |
| Diversity / gradient-based | Batch selection balancing informativeness with spread across the space | More computation; still no guarantee of subgroup coverage | Better for batch labelling |
| Hybrid + semi-supervised | Active queries plus unlabelled data folded back into learning | Added pipeline complexity | Demonstrated to reduce sampling-bias effects |
Evidence here is not one-sided: one empirical study found active set selection using posterior entropy from deep models robust to sampling biases and to query size and strategy choices, contrary to earlier literature — so the safest approach is to test uncertainty sampling on your own data rather than assume either result holds.
The compounding risk is worth naming plainly. Active acquisition assumes the labels it collects are sound, but many active-learning applications rely on human-generated labels that are highly bias-prone, and research into active data acquisition under label bias documents patterns in which more data leads models astray rather than improving them. Volume does not correct a biased selection rule; it entrenches it. One caution specific to multilingual programmes: a model used for pre-labelling almost always performs worse in lower-resource languages, so the same workflow that saves time in a high-resource language can quietly anchor annotators to poor suggestions elsewhere — which is why per-language model evaluation before enabling pre-labels, paired with native-speaker review, matters as much for low-resource speech data collection as it does for text.
In practice, five checks catch most of the failure modes above. Evaluate the pre-label model's error profile by class and subgroup before trusting it — a model that is 92% accurate overall can be 40% accurate on the cases that matter. Hold out a blind control set annotated from scratch, without suggestions, so anchoring has something to be measured against. Track the accept rate as a warning signal: a rate near 100% usually means annotators have stopped reviewing rather than that the model has become excellent. Keep a random slice in the mix alongside active queries to preserve a representative, unbiased evaluation set. Turn pre-labels off entirely for subjective tasks — sentiment, toxicity and preference judgements are where anchoring does the most damage and shows the least. These are the same layered checks behind layered quality control before delivery, applied one stage earlier, before the data ever reaches a customer-facing gold set — the discipline that also underpins AI data validation and dataset work built for enterprise LLM training.