Where Did This Data Come From?

For most datasets used to train AI, nobody can fully answer that question. Courts, regulators and the mathematics of model training are all beginning to require an answer.

A machine-learning engineer downloads a dataset from a popular hosting platform. The page says it is released under a permissive licence that allows commercial use. She fine-tunes a model on it and the model goes into a product.

How confident should she be in that licence?

In 2024, a group of researchers calling themselves the Data Provenance Initiative published an audit of more than 1,800 text datasets widely used for fine-tuning. They traced each one back to its original source and compared the licence its creators had specified with the licence shown on the hosting platforms where people find it.

On those platforms, licence information was missing for more than 70% of datasets. Where a licence was shown, it was miscategorised more than half the time, and the errors usually ran in one direction: the platform listed terms more permissive than the creators intended.

The engineer’s confidence should be low. Her situation is ordinary. Datasets are assembled from other datasets, repackaged, filtered, translated and re-uploaded, and at each step the record of where the content came from, and on what terms, tends to fall away.

What provenance means

Provenance is the documented history of a piece of data. For AI training data, it covers at least five things.

For most of the history of machine learning, hardly any of this was recorded. Geiger and colleagues reviewed a sample of machine-learning papers that used human-labelled data and found that many did not report basic facts such as who the annotators were, whether they were trained, or how agreement was measured. A widely cited proposal, datasheets for datasets, was written in response to this gap. It took its inspiration from the electronics industry, where every component ships with a document describing its characteristics and limits.

The absence of provenance has not been a theoretical concern.

Three datasets that came apart

LAION-5B is a collection of nearly six billion image–text pairs scraped from the web, used to train widely deployed image generators. In late 2023, an investigation by the Stanford Internet Observatory found that it contained links to more than a thousand instances of child sexual abuse material. The dataset was taken offline and later re-released after cleaning. Earlier academic work by Birhane and colleagues had already documented other harmful content in its predecessor. Nobody had reviewed the contents, because at that scale nobody could.

Retracted datasets continue to circulate. Peng and colleagues studied what happened to several face and person-recognition datasets after their creators withdrew them over consent and privacy concerns. The datasets, and derivatives of them, kept being used and cited in research for years. Withdrawal by the original source did not reach the copies.

The open web is closing. A second study from the Data Provenance Initiative audited 14,000 web domains behind major training corpora and documented a sharp rise in restrictions on AI crawlers through robots.txt files and terms of service. Content that was freely collected a few years ago is now explicitly withheld. Data scraped before the restriction and data scraped after it carry different positions on consent, and a dataset without dates cannot say which it contains.

Three pressures converging

Courts. Litigation over the use of copyrighted material in training has moved from filing to rulings. In the United States in 2025, a federal court in the case of Thomson Reuters against Ross Intelligence rejected a fair-use defence for the use of legal headnotes to train a competing legal research tool. In a case brought by authors against Anthropic, a federal judge held that training on lawfully acquired books was fair use, and drew a distinction for copies obtained from pirate libraries; the case later settled for a reported 1.5 billion dollars. In a case against Meta, another judge found for the company on the record before him while expressing doubt about the general proposition. The United States Copyright Office published a lengthy report on training and copyright in the same year.

The law is unsettled and differs between jurisdictions, and nothing here is legal advice. One practical point is common to the rulings. The outcome turned on facts about the data: how it was obtained, from where, and what was done with it. An organisation that cannot establish those facts is poorly placed in any dispute.

Regulators. The European Union’s AI Act imposes data governance obligations on high-risk systems under Article 10, including requirements about the origin of data, how it was collected and prepared, and examination for bias. For general-purpose models, providers must publish a summary of the content used for training, following a template the Commission issued in 2025. In India, the Digital Personal Data Protection Act of 2023 sets conditions on processing personal data, with implementing rules notified in 2025. The United States NIST AI Risk Management Framework, though voluntary, treats documentation of data provenance as part of responsible practice.

Each of these asks organisations to state where training data came from. That cannot be reconstructed after the fact.

The technical need. As the article on model collapse argued, the effect of synthetic data on a model depends on the proportion of real data in the mix. Managing that proportion requires knowing which records are human-made. Detection after the fact is unreliable. The only dependable method is to record origin when the data is created or acquired.

Provenance is also what makes debugging possible. When a model misbehaves, the question is which data taught it that. Without lineage, the question has no answer.

The cost of caring

There is a serious argument that strict provenance requirements do more harm than good.

Much of the progress in AI came from large, loosely documented, scraped datasets. Requiring full provenance would exclude most of them. It would favour large companies that can afford licensing and compliance over researchers and start-ups. And it would exclude data from communities and languages that are under-represented, where formal licensing is rare, making models less inclusive.

These are real costs. The Data Provenance Initiative’s own authors note that restrictions on crawling fall unevenly and may skew what models learn.

Two responses are available. First, provenance is not the same as permission. Recording where data came from does not by itself prevent its use. It enables an informed decision about risk. An organisation may choose to use data of uncertain status for research and not for a commercial product, and it can only make that distinction if it knows which data is which.

Second, the balance of risk depends on the use. For a research prototype, the consequences of poor provenance are small. For a product deployed to customers in a regulated industry, they include injunctions, forced retraining and regulatory penalties. The appropriate standard varies accordingly.

What a provenance record contains

Several documentation standards exist: datasheets for datasets, data statements for language data, data cards, and the dataset nutrition label. They differ in format and agree on substance. A usable record answers the following.

Origin. Where did each part of the data come from? Who created it, when, and for what purpose?

Acquisition. How was it obtained: collected directly, licensed, scraped, purchased, generated?

Rights. Under what licence or terms? What uses are permitted? Does it contain personal data, and on what basis was it collected?

Composition. What is in it? Which populations, languages and time periods are represented? What is known to be missing?

Processing. What filtering, cleaning, deduplication, translation or augmentation was applied?

Labelling. Who annotated it, with what qualifications, under what instructions, with what agreement and quality control? Was AI assistance used?

Synthetic content. Which records, if any, were machine-generated, by which model?

Lineage. Which other datasets were derived from this one, and which models were trained on it?

For commissioned data, all of this is available at the time of creation if somebody thinks to write it down. For acquired data, it has to be requested, and the supplier’s ability to provide it is a good indicator of the supplier.

Building it into the pipeline

Provenance documentation written once as a compliance exercise goes stale. Durable practice treats it as metadata carried with the data.

Record at ingestion. Every dataset entering the organisation gets a record of source, date, terms and contact before it is used.

Tag at the record level where it matters. Source, licence, date and human-or-synthetic origin, attached to individual records and kept through transformations.

Version datasets. A training run should reference a specific, immutable version, so that a model’s training data can be stated exactly.

Keep the link from model to data. For each model, which datasets and versions went into it. This is what a regulator, a litigant or an internal investigation will ask for.

Plan for removal. If a source must be withdrawn, because consent is revoked or a licence disputed, can you identify the affected records and the models trained on them?

Require it from suppliers. Make provenance documentation a contractual deliverable for any purchased or commissioned data.

What this changes