Stop Labeling Everything. Carefully.

Active learning promises the same model for a fraction of the labels. The research says the promise is real, conditional, and easier to break than most introductions admit.

Suppose you have a million unlabelled images and a budget to label ten thousand. Which ten thousand?

The default answer is a random sample. It is simple and unbiased, and it wastes most of the budget. If 95% of the images are easy cases that a model would get right after seeing a few hundred, then roughly 9,500 of the labels will teach it almost nothing.

Active learning is the alternative. Train a model on a small labelled seed. Use it to score the unlabelled pool. Send the most informative examples to annotators. Retrain, and repeat. The idea goes back to the early 1990s, and Burr Settles’s 2009 survey remains the standard reference for its many variants.

The core intuition is sound and the published gains can be large. It is also among the techniques with the widest gap between how often it is recommended and how reliably it works. That gap is worth understanding before building a labelling budget around it.

How the selection works

Most methods fall into two families.

Uncertainty sampling picks examples the current model is least sure about: those nearest its decision boundary, or where its predicted probabilities are most evenly spread. The logic is that a confident prediction is probably right and a label would confirm what the model already knows.

Diversity sampling picks examples that are unlike those already labelled, to cover the input space. The core-set approach of Sener and Savarese is the best-known formulation: choose points such that every unlabelled example is close to some labelled one.

Practical systems combine the two, selecting batches that are both uncertain and varied, since the hundred most uncertain examples are often a hundred near-copies of the same hard case.

Where it has worked

The clearest successes are in settings with three features: a large unlabelled pool, a heavily skewed distribution, and expensive labels.

Autonomous driving is the standard example. Fleets record far more footage than could ever be annotated, nearly all of it uneventful. Companies in the field have described mining that footage for the scenes their models handle worst, using model disagreement and uncertainty as triggers, and labelling those. The Segment Anything data engine described in the previous article applies similar logic: annotators were directed to the objects the model had missed.

Rare-class problems benefit for the same reason. If one transaction in a thousand is fraudulent, a random sample of ten thousand contains about ten positives. A model-guided sample can contain hundreds.

In these cases, the alternative to active selection is not a slightly less efficient random sample. It is a dataset that barely contains the phenomenon of interest.

Where it has not

In 2019, Lowell, Lipton and Wallace published a paper with a deliberately plain title: “Practical Obstacles to Deploying Active Learning”. They ran standard methods across several text classification and sequence labelling tasks, with several model types, and reported three findings that anyone planning to use the technique should know.

Gains were inconsistent. Active learning beat random sampling in some combinations of dataset and model and lost in others. There was no way to know in advance which would occur, and a practitioner cannot run both to find out, because the point is to label only once.

The data is tied to the model that chose it. A dataset selected by one model was often no better than random, and sometimes worse, when used to train a different model. This matters because the labelled data usually outlives the model. Teams change architectures every year or two. A dataset shaped by the uncertainties of the previous one carries its blind spots forward.

The sample is no longer representative. An actively selected set over-represents hard and unusual cases. It cannot be used to estimate real-world performance, class frequencies or anything else that depends on the natural distribution.

Other studies add to the list. Karamcheti and colleagues investigated why active learning underperformed on visual question answering and traced it to what they called collective outliers: groups of examples that the model is highly uncertain about because they are unanswerable, mislabelled or require knowledge the model cannot acquire. Uncertainty sampling sought these out avidly. The budget went to examples that could not be learned from.

Munjal and colleagues re-ran a range of deep active-learning methods under controlled conditions and found that many reported improvements over random sampling shrank or vanished once the random baseline was properly tuned and regularised. Part of the published advantage had been an artefact of weak baselines.

There is also a cold-start problem. In the first rounds, the model has seen too little to have meaningful uncertainty. Its selections are close to noise, and can be worse than random because they are biased noise.

Reconciling the two pictures

How can the same technique be indispensable in one setting and unreliable in another?

The successes share a property. In driving and fraud, the natural distribution is so skewed that almost any intelligent selection beats random. The gain comes from finding rare things, and it survives a change of model because the rare things remain rare and relevant whatever the architecture.

The failures cluster in balanced academic benchmarks, where most examples carry some information and random sampling is already a reasonable strategy. There, the gain depends on fine distinctions about which examples are marginally more useful to a particular model, and those distinctions do not transfer.

The practical reading is that active learning is most trustworthy when it is used as a search tool for under-represented cases, and least trustworthy when it is used as a fine optimiser of labelling efficiency on data that is already reasonably balanced.

Using it without getting hurt

A few design choices address most of the documented failure modes.

Always keep a random stream. Label a random sample alongside the selected one, perhaps a fifth to a third of the budget. It provides an unbiased evaluation set, an estimate of the true distribution, a guard against the selector’s blind spots, and a baseline that shows whether selection is helping.

Never evaluate on selected data. The test set must come from the natural distribution, or from the deployment environment directly. A model evaluated on its own hardest cases will look worse than it is. One evaluated on a set chosen by a previous model will look arbitrary.

Filter before you send. Examples flagged as most uncertain should pass a quick check for being answerable and in scope. Route the remainder to a separate queue: some will be labelling errors or guideline gaps, which are worth knowing about for other reasons.

Combine uncertainty with diversity. Batches selected purely on uncertainty are redundant. Clustering candidates and sampling across clusters is a simple and effective correction.

Start with a random or diverse seed. Do not use model-guided selection until the model is good enough for its uncertainty to mean something.

Select with an ensemble or a strong general model. Disagreement among several different models is a more architecture-neutral signal than one model’s confidence. Using a large pretrained model’s embeddings for diversity selection also reduces dependence on the task model. Coleman and colleagues showed that a small, cheap proxy model can do the selecting for a larger one with little loss, which helps with compute cost though not with the transfer problem in general.

Use production as the selector. The most reliable source of informative examples is failure in deployment: user corrections, low-confidence predictions on real inputs, disagreements between the model and a human reviewer. This is active learning in the sense that matters, and it selects from the distribution the model will keep facing.

The economic argument, stated properly

Active learning is usually sold on cost reduction: the same accuracy with fewer labels. Given the evidence, a more defensible claim is about allocation. A fixed budget buys more useful information when part of it is directed.

That reframing matters for how the technique interacts with the rest of an annotation operation. Selected examples are harder, so they take longer and produce more disagreement. An operation priced per item, staffed for average difficulty, will see throughput fall and error rates rise when active learning is switched on, and may conclude the method has failed when it is working as designed. The hard items are where expert annotators and adjudication, discussed in earlier articles, should be concentrated.

What this changes