“High Quality” Is Not a Property of Data

A dataset cannot be good in general. It can only be good for a purpose, and most quality metrics never ask which one.

In 2018, Joy Buolamwini and Timnit Gebru evaluated three commercial systems that classified gender from photographs of faces. For lighter-skinned men, the worst error rate among the three was 0.8%. For darker-skinned women, error rates ran as high as 34.7%.

The labels in the underlying training and benchmark data were not the issue. A photograph labelled as a man was, as far as anyone could tell, a photograph of a man. The issue was who was in the photographs. When the authors examined two standard face benchmarks of the time, they found that 79.6% of the faces in one and 86.2% in the other were lighter-skinned.

By the most common definition of data quality, the percentage of labels that are correct, those datasets were excellent. By any definition that mattered to a darker-skinned woman being classified by the resulting system, they were poor.

This is the central difficulty with the phrase “high-quality training data”. It is used as though quality were intrinsic to the data, like weight. It is not.

An older and better definition

The data-management field settled this question before modern machine learning existed. In 1996, Richard Wang and Diane Strong published a study that asked data consumers, not data producers, what quality meant to them. Their definition has been standard since: data quality is fitness for use.

Two things follow from that definition. First, quality cannot be assessed without knowing the use. Second, accuracy is only one of several things users care about. Wang and Strong identified fifteen dimensions in four groups: intrinsic qualities such as accuracy and believability, contextual qualities such as relevance, completeness and timeliness, representational qualities such as consistency and interpretability, and accessibility.

Machine learning rediscovered the same idea through failure. The pattern in the previous article in this series, in which a model validates well and then breaks in deployment, is what fitness for use looks like when it is absent.

Six questions that define quality for training data

Translating the idea into something a team can apply, the research literature supports at least six separate questions. A dataset can score well on some and badly on others.

Are the labels correct? This is the dimension everyone measures. It matters, and methods such as confident learning and multi-annotator review address it. But it is the dimension on which most professionally produced datasets are already adequate.

Does it cover the conditions of use? Coverage means the range of inputs the model will meet is present in sufficient number: the lighting, the accents, the document layouts, the patient populations. The WILDS benchmark, assembled by Koh and colleagues from ten real-world datasets, documents consistent performance drops when models meet hospitals, regions, cameras or time periods that were absent from training. Coverage failures are invisible to accuracy audits because the examples that would reveal them are not there to audit.

Are subgroups represented where it counts? Coverage in aggregate is not enough. Hidden stratification, described by Oakden-Rayner and colleagues, occurs when a clinically important subtype is rare within a class and the model fails on it while the class-level metric looks fine.

Is it labelled consistently? Two annotators applying the same guidelines should produce compatible labels, and the same annotator should do so over time. Inconsistent labelling is different from wrong labelling: each individual label may be defensible, while the dataset as a whole teaches the model a boundary that moves.

Do we know where it came from? Provenance covers source, collection method, consent, licence and processing history. A large audit of more than 1,800 text datasets by the Data Provenance Initiative found licence information omitted for more than 70% of datasets on popular hosting platforms and miscategorised for more than half. Data of unknown origin carries legal exposure and, less obviously, technical risk: without knowing how it was filtered, nobody can say what is missing.

Is it still true? Language, products, fraud patterns and road layouts change. Lazaridou and colleagues showed that language models degrade measurably on text from periods after their training data ends. A dataset is a photograph of a distribution that keeps moving.

Why accuracy dominates anyway

If quality has six dimensions, why do most specifications mention one?

Partly because accuracy is easy to measure. You sample labelled items, have someone check them, and report a percentage. The other dimensions require knowing something about deployment, which the people producing the data often do not.

Partly because of how the work is bought. Annotation is commonly procured against an accuracy threshold in a service agreement. Coverage is the client’s responsibility, defined by whatever raw data the client hands over. The vendor is accountable for labelling what it was given, correctly. Nobody is accountable for whether what was given was the right thing to label.

And partly because the research culture long treated datasets as fixed. Benchmark datasets were inputs to be downloaded, not artefacts to be examined. The data-centric AI movement, and benchmarks such as DataPerf that hold the model fixed and score improvements to the data, are a deliberate correction.

Is this unmeasurable?

The obvious objection is that “fitness for use” is a slogan, not a metric. Accuracy at least produces a number. If quality depends on context, every dataset needs a bespoke assessment and nothing can be compared or contracted.

The objection is weaker than it looks, because most of the dimensions can be measured once the use is stated.

Coverage can be measured by defining the slices that matter, such as device type, language, region or demographic group, and counting examples in each. Subgroup adequacy can be measured by evaluating on each slice separately. Consistency can be measured by inter-annotator and intra-annotator agreement. Provenance can be measured by completeness of documentation. Freshness can be measured by comparing the distribution of recent production inputs against the training set.

Tooling for parts of this is mature. Google’s data-validation work for production pipelines, and Amazon’s equivalent, check incoming data against an expected schema and distribution automatically. Documentation standards such as datasheets for datasets and data cards give a structure for recording the rest. An international standard series for data quality in analytics and machine learning, ISO/IEC 5259, now exists as well.

What cannot be avoided is the first step: someone has to say what the model is for, where it will run and on whom. That is uncomfortable because it is a product decision, not a data decision. But a dataset specification that omits it is not neutral. It has assumed that deployment will resemble whatever data happened to be available.

The limit of the argument

Fitness for use should not become an excuse for ignoring label accuracy. In domains where labels are hard, such as medical imaging or legal classification, accuracy may well be the binding constraint, and it is the subject of several later articles in this series.

There is also a real tension between dimensions. Maximising coverage of rare subgroups usually means collecting data from harder sources, which lowers label accuracy and agreement. Insisting on full provenance excludes data that would have improved coverage. Quality is a set of trade-offs, and a team that claims to have maximised every dimension has not looked closely at any of them.

Writing a better specification

A workable data specification answers the six questions explicitly.

It states the deployment conditions and the slices that matter. It sets minimum counts per slice, not only a total. It sets an accuracy target and says how it will be measured and by whom. It sets an agreement target, and says what happens to items on which annotators disagree. It requires documentation of source, consent and processing. And it sets a review date, after which the data is assumed stale until checked.

Most of that fits on a page. Writing the page is the hard part, because it forces the conversation about purpose that the word “quality” was being used to avoid.

What this changes