Disagreement Is Data

When two careful people label the same item differently, the usual response is to pick a winner. The evidence says that often throws away the most useful information in the dataset.

Take a sentence pair of the kind used in natural-language inference tasks. The premise: a man is standing outside a café holding a cup. The hypothesis: the man has bought a coffee. Does the first statement imply the second?

Ask one hundred people and you will not get one answer. Some will say yes, because that is the obvious reading. Some will say it is undetermined, because the cup might hold tea, or might be empty, or might belong to someone else. Both groups are reading carefully. They differ on how much everyday inference the task permits.

The conventional pipeline handles this by collecting three or five labels, taking the majority, and recording a single “gold” answer. The minority view is discarded as noise.

In 2019, Ellie Pavlick and Tom Kwiatkowski tested whether that was justified. They collected many more judgements per item than usual on inference datasets and looked at the distributions. For a substantial share of items, disagreement did not shrink as more annotators were added. The judgements clustered around two different answers. They also found that supplying more context did not make the disagreement go away. Their conclusion was that the disagreement was a property of the items, not of careless annotators.

A follow-up study, ChaosNLI, collected one hundred labels each for several thousand examples from popular inference benchmarks and found the same thing at scale: many items have a spread of human opinion, and models trained on single labels were poor at predicting that spread.

Four reasons people disagree

“Disagreement” covers different phenomena. Separating them is the first practical step, because each has a different remedy.

Error. Someone was tired, rushed or misread the item. The label is wrong and they would agree if shown. Redundancy and review fix this.

Underspecified guidelines. The instructions did not say whether a partially hidden pedestrian counts, or whether sarcasm is “negative”. Annotators filled the gap differently. Each label is reasonable under some reading of the instructions. The fix is to the guideline, and re-labelling without changing it reproduces the split.

Ambiguity in the item. The café sentence. The item supports more than one reading and no guideline of reasonable length would settle every such case. The honest label is a distribution.

Difference in perspective. The annotators bring different knowledge, experience or values, and the task depends on them. Whether a remark is offensive is the standard example.

The first two are defects in the process. The last two are information about the world.

What the perspective evidence shows

The research on subjective tasks is by now extensive, and uncomfortable for anyone who assumes a neutral annotator exists.

Sap and colleagues showed in 2019 that tweets written in African American English were substantially more likely to be labelled offensive by annotators than other tweets, and that models trained on those labels reproduced the pattern. When annotators were primed to consider the dialect of the author, the rate fell. The labels had encoded the annotators’ unfamiliarity as a property of the text.

A later study by the same group, “Annotators with Attitudes”, measured annotators’ own beliefs and identities and found that they predicted toxicity ratings in systematic ways. Annotators who scored higher on measures of racist beliefs were less likely to rate anti-Black language as toxic and more likely to rate African American English as toxic. Work at Google on rater identity found comparable effects: raters who belonged to a group targeted by a comment rated it differently from those who did not.

In these settings, majority vote does not converge on truth. It converges on the view of whichever group supplies the majority of annotators.

Medicine shows a structurally similar pattern for a different reason. Studies of diabetic retinopathy grading found meaningful variation between individual ophthalmologists reading the same image. Here the disagreement reflects difficulty and expertise, and the remedy that worked was adjudication: specialists discussing cases until they reached a consensus that was better than any individual vote.

What to do with a split

Aroyo and Welty named the underlying assumption in a 2015 paper whose title is its argument: “Truth Is a Lie”. They listed seven myths of annotation, among them that each item has one correct answer, that disagreement is bad, and that more detailed guidelines always help. Barbara Plank later proposed a term now widely used, human label variation, to describe disagreement that should be modelled and not removed.

Several practical methods follow.

Keep the individual labels. The cheapest change is to stop discarding them. A dataset that stores every annotator’s judgement, with an annotator identifier, can support majority vote and everything else. One that stores only the aggregate cannot be un-aggregated.

Train on distributions. Peterson and colleagues collected about fifty human judgements each for ten thousand CIFAR-10 test images and trained classifiers on the resulting distributions instead of single labels. The models generalised better to new datasets and were more robust to adversarial perturbation. Where humans were unsure, the model learned to be unsure, and that turned out to be useful.

Model annotators, not only items. Davani, Díaz and Prabhakaran trained models that predicted each annotator’s label separately before combining them. Performance matched or beat majority-vote training, and the model’s internal spread gave a better estimate of when it was likely to be wrong.

Evaluate against the spread. If the test set forces a single answer onto items where humans split sixty-forty, forty per cent of the “error” assigned to a model is disagreement with a coin flip. Evaluating on the distribution, or separately on high- and low-agreement items, gives a truer picture.

The case for a single answer

There are good reasons the single-label convention persists.

Many downstream systems need a decision. A content-moderation queue removes a post or leaves it. A triage model flags a scan or does not. A distribution over labels has to be collapsed at some point, and someone has to choose the threshold.

Collecting enough labels per item to estimate a distribution is expensive. Five annotators cost five times what one costs, and most of the time, on most items, they agree.

And embracing disagreement can become an excuse. A team with sloppy guidelines can declare its noise to be “human label variation” and avoid the work of fixing them. Not every split is meaningful. Sometimes one annotator is simply wrong.

These objections argue for discipline, not for discarding the idea. The decision has to be made eventually, but it is better made by a system that knows which cases are contested. Redundant labelling can be targeted: collect more judgements on items where the first two annotators disagree, and stop early where they concur. And the way to tell error from ambiguity is to look, which is the purpose of the taxonomy above.

A working procedure

For teams that want to act on this without rebuilding their pipeline, a modest version is enough.

First, pilot the guideline on a small sample with several annotators, and read the disagreements. Most early disagreement is the second type, underspecification, and a round or two of guideline revision removes much of it.

Second, once the guideline is stable, classify the disagreement that remains. If annotators converge after discussion, it was error or oversight. If they understand each other and still differ, it is ambiguity or perspective.

Third, decide by design what the dataset should represent. For a factual extraction task, the answer may be adjudicated truth. For toxicity, it may be the distribution of views across a defined population, with the population specified and the annotator pool recruited to match it.

Fourth, record who the annotators were, in aggregate. A dataset of subjective judgements without a description of who judged is missing its most important metadata.

What this changes