The Expert Premium

A specialist’s hour costs many times a generalist’s. Whether that is expensive depends entirely on what the generalist’s label is worth.

To train one of the first deep-learning systems for detecting diabetic eye disease, a Google team needed labels for 128,175 retinal photographs. They did not use a crowd. They used a panel of 54 ophthalmologists and senior ophthalmology residents licensed in the United States, and had each image graded between three and seven times.

Published in JAMA in 2016, the resulting system showed high sensitivity and specificity for referable disease on two validation sets. It is among the most cited results in medical AI. It is also an annotation project of unusual cost: several hundred thousand individual judgements by physicians.

Eight years earlier, a different and equally cited study had reached what looks like the opposite conclusion. Snow and colleagues showed that for several language tasks, four non-expert workers recruited online could match the label quality of one expert, at a small fraction of the price.

Both results are sound. The interesting question is what separates the tasks in the first study from the tasks in the second, because that boundary determines the economics of every data budget in a specialised field.

Three kinds of task

A useful way to sort annotation tasks is by what a non-expert lacks.

Nothing. Is this headline positive or negative? Is there a car in this image? Any attentive adult has the knowledge required. Individual error comes from inattention, and inattention is roughly random, so redundancy cancels it. This is the regime of Snow’s result.

Training. Draw a box around every pedestrian, including partly hidden ones, following a forty-page specification. The required knowledge is the specification itself. It can be taught in days, and performance then depends on diligence and practice. Most commercial annotation sits here.

Years. Does this retinal image show moderate or severe non-proliferative retinopathy? Is this contract clause an uncapped indemnity? Is this code change safe to merge? The required knowledge cannot be written into guidelines of any practical length, because it is the professional training itself.

In the third category, the logic of redundancy breaks. Four non-experts do not average out to one expert, because their errors are not random. They are the same error: the one that comes from not knowing. As the previous article showed, majority voting amplifies shared mistakes.

The evidence from three professions

Medicine. The Google retinopathy work was followed by a study of the labels themselves. Krause and colleagues found that even among ophthalmologists, individual graders varied considerably, and that an adjudicated consensus of retinal specialists identified disease that a majority vote of individual graders missed. Expertise turned out to have levels, and the reference standard mattered at each one. Work on chest radiography by Majkowska and colleagues took the same approach, building reference standards through adjudication by panels of radiologists because single-reader labels were not reliable enough to evaluate against.

Law. The Contract Understanding Atticus Dataset was built to train models to find important clauses in commercial contracts. Its more than 13,000 annotations were produced by dozens of legal experts, and its authors describe the cost of expert annotators as the main bottleneck for specialised domains. LegalBench, a later benchmark of 162 legal reasoning tasks, was assembled through contributions from legal professionals for the same reason: nobody else could write the tasks or verify the answers.

What happens without that expertise in the loop is now documented. A Stanford study led by Magesh evaluated commercial AI legal research tools marketed as reliable, and found they produced incorrect or misgrounded answers between 17% and 33% of the time. Detecting those errors required lawyers reading each response against the cited authorities.

Finance. FinanceBench tested language models on questions about public company filings, with answers and supporting evidence annotated by hand. A leading model equipped with a retrieval system answered incorrectly or declined to answer 81% of a sample of questions. Again, establishing that required someone who could read a filing and know the right answer.

In each field, expertise is needed twice: once to create training signal, and again, more stringently, to evaluate whether the model learned it.

The yield calculation

The standard objection to expert annotation is price. A specialist’s time costs many multiples of a generalist’s. On a cost-per-label basis the comparison looks hopeless.

Cost per label is the wrong unit. The quantity that matters is cost per correct label, which depends on yield.

Consider a task where a generalist is right 60% of the time and wrong in a consistent direction on the rest. Three generalist votes do not rescue it. The 40% of items on which they are wrong are items where they tend to be wrong together. The dataset is delivered at low cost, with a large fraction of labels incorrect and no indication of which.

The downstream costs then arrive. The model learns the systematic error. Evaluation, if it uses labels from the same pool, cannot detect it. The problem surfaces in deployment, as the cascades article described, and the remedy is to re-label with experts anyway, having also paid for the first attempt and the failed model.

On tasks of the first and second kind, generalists have a yield close to that of experts and the cheap option is correct. On tasks of the third kind, the cheap option’s yield can be low enough that its true cost per usable label exceeds the expert’s.

Making expertise go further

Expert time is scarce as well as expensive. In many health systems there are too few radiologists to read clinical scans, let alone to label training data. The practical problem is allocation, and several designs help.

Tiering. Trained generalists handle the full volume and flag items they find uncertain or that fall into predefined hard categories. Experts see only the flagged subset. This works when generalists know what they do not know, which has to be verified, not assumed.

Adjudication, not labelling. Experts are most valuable resolving disagreements, as the retinopathy studies showed. Two trained annotators label independently, and an expert sees only the items where they differ.

Experts write, others apply. Much of an expert’s knowledge can be partially transferred into guidelines, worked examples and decision trees. An hour of a specialist’s time spent improving a guideline can raise the quality of thousands of generalist labels. An hour spent labelling produces a few dozen.

Reserve experts for evaluation. As the article on test sets argued, evaluation data is small and deserves the highest standard. If the expert budget is limited, the test set is the place to spend it first.

Use the model to route. Active learning and uncertainty estimates can direct expert attention to the examples a model finds hardest. A later article in this series examines how well that works.

Where the premium is overpaid

The argument can be pushed too far, and vendors of expert annotation have reasons to push it.

Credentials are not skill at annotation. A senior physician may be an inconsistent labeller, impatient with guidelines and prone to applying personal conventions. Expert disagreement is real, as Krause’s work shows, and a single expert label is not ground truth. Expert panels need the same agreement monitoring and adjudication as any other annotators.

Many tasks in specialised fields are not specialised tasks. Extracting a patient’s date of birth from a clinical note does not need a clinician. Classifying a legal document as a lease or a licence may not need a lawyer. Decomposing a workflow often shows that a small share of the decisions require expertise and the rest require care.

And the boundary moves. As models improve, tasks that needed training can be pre-labelled, and tasks that needed experts can be drafted for expert review. The premium does not disappear, but it concentrates on a smaller and harder set of judgements.

A test before you commit

Before commissioning a large annotation project in a specialised domain, a small experiment settles most of the question.

Take two hundred representative items. Have them labelled by trained generalists and, independently, by two or three qualified experts with adjudication. Compare.

If generalists match the adjudicated expert labels at a high rate, the task is of the first or second kind and expert annotation is unnecessary for the bulk of it. If they diverge, look at where. Random scatter suggests a training or guideline problem. A consistent pattern on particular item types identifies exactly which items need an expert. The experiment costs little, and it replaces an argument about hourly rates with a measurement of yield.

What this changes