The Arithmetic of Trust

Majority voting rests on a two-century-old theorem with two assumptions. Annotation pipelines violate both, and the mathematics of what to do about it has existed since 1979.

In 1785 the Marquis de Condorcet proved a theorem about juries. If each juror is more likely than not to reach the correct verdict, and jurors decide independently, then the probability that the majority is correct rises toward certainty as the jury grows.

Modern data annotation runs on this theorem, mostly without knowing it. Collect three labels per item, take the majority, call it ground truth. The reasoning is Condorcet’s: individuals err, crowds converge.

The arithmetic is worth seeing, because it shows both the strength of the idea and where it stops.

Three annotators who are each right 80% of the time produce a correct majority 89.6% of the time. Five produce 94.2%. Redundancy works, and it works quickly.

Now look at the top row. Annotators who are right 60% of the time need nine votes to reach 73%. And at any level of individual skill, the calculation assumes that annotators’ mistakes are independent. If they all misread the same guideline, or lack the same expertise, or accepted the same machine-generated suggestion, adding more of them adds confidence without adding accuracy.

Condorcet’s theorem also has a less quoted corollary. If each juror is worse than chance, the majority converges on the wrong answer.

Not all annotators are equal

In 1979, two statisticians, Philip Dawid and Allan Skene, addressed a problem from clinical medicine. Several clinicians had each recorded a patient’s condition, their records disagreed, and there was no independent way to know who was right. Simple voting treated every clinician as equally reliable. Dawid and Skene proposed something better.

Their model gives each rater a confusion matrix: the probability that this rater says each category, given the true category. The true labels are unknown, but the model can be fitted by alternating two steps. Given a current guess at the true labels, estimate each rater’s confusion matrix. Given those matrices, re-estimate the true labels, weighting each rater by their estimated reliability. Repeat until stable.

The result is an estimate of the most likely true label for every item, and an estimate of every rater’s pattern of error, with no ground truth required.

Three ideas in that paper still organise the field.

Reliability is per annotator. A vote from a careful expert and a vote from someone guessing should not count the same.

Reliability is per category. An annotator may be excellent at identifying one class and poor at another. A confusion matrix captures that; a single accuracy score does not.

Bias is recoverable. This is the least intuitive and most useful point. Suppose an annotator consistently labels “severe” whenever the truth is “moderate”. Their accuracy is poor. But their labels are highly informative, because the error is predictable and can be inverted. Ipeirotis, Provost and Wang developed this distinction for crowdsourcing: an annotator who is biased but consistent is far more valuable than one who is unbiased but random, and simple accuracy scores cannot tell them apart.

What the crowdsourcing literature added

The arrival of online labour markets in the 2000s turned Dawid and Skene’s clinical problem into an industrial one, and produced a large body of evidence.

Snow and colleagues, in a much-cited 2008 study, compared non-expert labels from Amazon Mechanical Turk with expert labels across five language tasks. For tasks such as rating the emotional tone of headlines, an average of four non-expert labels per item matched the quality of an expert’s. The paper’s title asked whether cheap and fast could also be good, and for those tasks the answer was yes.

Sheng, Provost and Ipeirotis examined the trade-off between labelling more items and re-labelling the same ones. When individual labels are noisy, they found, collecting additional labels for existing items can improve a model more than collecting new items, and selectively re-labelling the items most likely to be wrong is better still.

Later models added more structure. Whitehill and colleagues modelled item difficulty alongside annotator ability, since a disagreement on a hard item says less about an annotator than a disagreement on an easy one. Raykar and colleagues trained the classifier and the annotator model jointly, so that the model’s own predictions helped decide whom to trust.

A sober note comes from a 2017 benchmark by Zheng and colleagues, which compared seventeen truth-inference algorithms across several real crowdsourced datasets. Its title asked whether the problem was solved, and the answer was no: no algorithm was best across all datasets, and on some tasks plain majority voting was competitive with far more elaborate methods. When annotators are of similar quality and enough labels are collected, the sophistication buys little. The models earn their keep when quality is uneven, redundancy is thin or errors are systematic.

When the independence assumption fails

Every method above still assumes that, given the true label, annotators err independently. Three developments have made that harder to believe.

Shared training. Professional annotation teams are trained together on the same guidelines by the same leads. That is good for consistency and bad for independence. Their errors are correlated by design.

Shared pre-labels. When a model proposes a label and humans confirm it, several humans confirming the same proposal are not several independent opinions.

Shared tools. In 2023, Veselovsky, Ribeiro and West re-ran a text summarisation task on Mechanical Turk and estimated, using keystroke patterns and a text classifier, that between 33% and 46% of crowd workers had used a large language model to produce their answers. Workers who paste a task into the same chatbot will agree with one another closely. A redundancy scheme would read that as high confidence.

The implication is that redundancy now needs diversity to be worth anything. Three labels from three annotators with different backgrounds, working without a shared suggestion, are stronger evidence than five labels from a homogeneous team correcting the same pre-label.

What voting has going for it

Weighted aggregation has costs that its advocates understate.

It is harder to explain. “Three people looked and two said yes” is something a client, an auditor or a regulator can follow. “The posterior probability under a fitted latent-class model is 0.83” is not.

It needs data. Estimating a confusion matrix for each annotator requires each annotator to have labelled enough items that overlap with others. In a fast-moving project with a rotating workforce, the estimates can be too noisy to help.

And it can entrench the majority. An unsupervised model learns reliability from agreement, so an annotator who is right when the group is wrong, perhaps the only domain expert in the pool, can be scored as unreliable and down-weighted. Without some ground truth to anchor the model, it measures conformity.

The last point is the reason no production system should rely on aggregation alone.

A practical trust system

In practice, good operations combine several signals.

Gold items. A fraction of each annotator’s queue consists of items with known answers, established by experts. This gives a direct measure of accuracy and anchors everything else.

Per-annotator, per-class statistics. Track confusion patterns, not only overall accuracy. An annotator who systematically confuses two categories needs targeted retraining, and their past labels in those categories can be flagged for review.

Adaptive redundancy. Send each item to two annotators. If they agree, accept it. If they disagree, send it to a third, or to a senior reviewer. This concentrates the labelling budget on the hard items, which is where it changes outcomes.

Diversity by design. For the items that get multiple labels, route them to annotators who are unlikely to share errors: different teams, different sites, with pre-labels hidden from at least one.

Weighted aggregation where the data supports it. With enough overlap and a gold anchor, a Dawid–Skene-style model will outperform voting. Use it there.

What this changes