Cost per Label Is the Wrong Unit

Annotation is bought by the item. Its value is realised by the model. Almost every bad purchasing decision in AI data comes from confusing the two.

An AI team needs 100,000 documents classified. Two suppliers respond. The first quotes eight cents per document. The second quotes twenty.

On a spreadsheet, the decision is made: 8,000 dollars against 20,000. In most procurement processes, a difference of that size would need a strong justification to overcome, and “quality” is hard to put a number on. The cheaper supplier wins.

The team finds out what it bought about three months later, when the model trained on those labels underperforms in testing and somebody starts looking at the data.

The spreadsheet compared prices. The quantity that should have been compared is the cost of obtaining a correct, useful training example, and that depends on things the per-item price does not show.

What a label costs

Start with a definition. The effective cost of one usable label is the total spent, divided by the number of labels that turn out to be correct and useful.

Written out, the numerator contains more than the unit price. It includes the redundancy needed to reach acceptable confidence, the quality assurance needed to find the bad labels, the internal engineering time spent specifying, checking and cleaning, and the cost of rework. The denominator is not the number of items delivered. It is the number that are right.

The table works through an illustration. The figures are hypothetical and chosen to show the mechanism. Supplier A’s labels are right 80% of the time on this task, so the buyer triple-labels, takes a majority and spends engineering time auditing the result. Supplier B’s are right 96% of the time with single labelling and a light audit.

On these assumptions the cheaper supplier costs more per correct label. The specific outcome depends on the numbers. The point is that the ranking can reverse, and the unit price contains no information about whether it will.

Sheng, Provost and Ipeirotis formalised part of this in 2008. They showed that when labels are noisy, a fixed budget can be better spent re-labelling existing items than acquiring new ones. The implication for pricing is direct: a noisy label is not a cheaper version of a good label. It is a fraction of one, and several must be bought to make a whole.

The cost that arrives later

Redundancy and rework are the visible part. The larger costs come after the labels have been accepted.

The model is worse than it should be. Label noise, particularly systematic noise, reduces accuracy. The team’s response is usually to try more model work: a larger architecture, more tuning, more compute. Engineering time is spent compensating for a data problem that has not been diagnosed as one.

The evaluation is wrong. If test labels come from the same source, the model’s measured performance is unreliable, as the article on label errors described. Decisions about what to ship rest on a faulty instrument.

The failure appears in production. The cascades research documented how data problems surface late and compound. By the time one is traced to its origin, the cost includes re-collection, re-labelling, re-training, re-validation and whatever harm occurred in between. In Sambasivan’s study, practitioners described abandoning entire datasets.

The engineering team becomes the quality department. Sculley and colleagues’ analysis of technical debt in machine-learning systems noted that the model code is a small part of a real system, surrounded by data collection, verification and monitoring. When purchased data is unreliable, expensive engineers end up doing data cleaning, the most costly way to do it.

None of these appears on the invoice. All of them are caused by what was on it.

Why low prices are low

It helps to ask how a supplier arrives at a very low unit price. Annotation cost is mostly labour. There are a limited number of ways to reduce it.

Pay people less. Research on crowd work has documented the result. An analysis of millions of tasks on Amazon Mechanical Turk by Hara and colleagues estimated median hourly earnings of about two dollars, with unpaid time spent searching for tasks and on rejected work accounting for much of the shortfall. Ethnographic studies, including Gray and Suri’s book-length account, describe the instability this creates. Low pay and high turnover mean annotators do not accumulate experience on a task, and experience is where quality comes from.

Give them less time. A per-item price implies a time per item. If it is too short for the judgement the task requires, annotators will take the shortcut the task allows.

Reduce oversight. Quality assurance is overhead. So are training, guideline development, pilot rounds and feedback to annotators.

Use automation quietly. Pre-labelling with a model and light human confirmation lowers cost considerably. As discussed earlier in this series, it also changes the error profile from random to systematic unless verified carefully.

Each of these is a reasonable way to reach a price. Each reduces yield on tasks that require care. The buyer who selects on price is selecting for them.

When cheap is correct

The argument should not be read as “always pay more”. For a large class of tasks the low-price option is the right one.

Where a task is simple, well specified and tolerant of random error, non-expert labels at low cost with modest redundancy perform well. Snow and colleagues’ result, that four inexpensive labels matched an expert for certain language tasks, has been replicated many times. Paying a premium for such work buys nothing.

Where volume matters more than precision, as in pretraining or weak supervision, noisy labels at scale can be the better use of a budget. The article on data volume set out the conditions.

And where a model can do the first pass reliably, verified automation is cheaper than either supplier.

The error is to apply one purchasing logic to all tasks. The useful question is what the task requires, and it is answered by measurement.

How active learning and tiering change the sum

Two techniques from earlier articles shift the economics in ways per-label pricing obscures.

Active learning reduces the number of labels needed by directing effort to informative examples. Those examples are harder, so each costs more. A supplier quoting a flat rate will lose money on an actively selected set, or will cut corners. The total cost falls while the unit cost rises, and a buyer focused on unit cost will see only the second.

Tiering directs easy items to generalists and hard items to experts. The blended price is higher than an all-generalist rate and far lower than an all-expert one. Its value depends on the routing being accurate, which is a capability to pay for.

In both cases the efficient design has a higher price per item and a lower cost per unit of model improvement.

A better way to buy

Several changes to procurement practice follow.

Run a paid pilot on your own task. Give each candidate supplier the same few hundred items, with a subset for which you hold expert-verified answers. Measure accuracy against them, by category. This costs little and produces the one number the price list omits.

Compute effective cost. With measured accuracy, estimate the redundancy and audit effort needed to reach your quality target with each supplier. Compare totals.

Price by difficulty. Agree different rates for different item classes. A flat rate rewards speed on hard items, which is the opposite of what you want.

Contract on measured quality. Specify accuracy against an agreed reference set, with defined sampling, in place of or alongside a price. Include what happens when it is missed.

Ask how the price is achieved. Annotator pay, training, tenure, time allowed per item, the extent of automation and how it is verified. These are the inputs to yield.

Account for internal time. Hours your engineers spend writing guidelines, answering questions and cleaning output are part of the cost of the data. A supplier who needs less of them is cheaper than the invoice shows.

Value evaluation data separately. A test set is small and its errors are costly. Apply a different standard and a different budget.

What this changes