Model Collapse, Read Carefully

A 2024 paper in Nature showed that models trained on their own output degrade over generations. What the result implies depends on an assumption that follow-up work has examined closely.

The experiment is simple to describe. Train a language model on human-written text. Use it to generate a new corpus. Train a second model on that corpus alone. Use the second model to generate a third corpus, and repeat.

In 2024, Shumailov and colleagues reported in Nature what happens. Over successive generations, the models degrade. First they lose the tails of the distribution: rare words, unusual constructions, minority patterns. The authors call this early collapse. Later, the output converges on a narrow band with little resemblance to the original data, which they call late collapse. In one demonstration, a model prompted with text about medieval architecture was, several generations on, producing repetitive text about jackrabbits.

The paper gave the phenomenon a memorable name, model collapse, and it was widely reported as a prediction that AI systems would poison themselves as the internet filled with machine-written text.

That reading goes beyond what the experiment shows. The follow-up research is more precise, and more useful to anyone making decisions about data.

Why it happens

The mechanism is statistical and not specific to any architecture.

A model trained on a finite sample cannot represent rare events as well as common ones. When it generates, the rare events are slightly under-produced. The next model, trained on that output, sees even fewer, and represents them less well again. Each generation samples from an approximation of the previous one, and the errors compound in one direction: toward the centre of the distribution.

Other groups observed the same effect in images. Alemohammad and colleagues trained generative image models in what they called an autophagous, or self-consuming, loop and found that without enough fresh real data in each generation, quality or diversity progressively declined. They named the condition Model Autophagy Disorder.

The qualifier in that sentence matters: without enough fresh real data.

The assumption that decides the outcome

The original experiment discards the previous data at each step. Generation three is trained only on the output of generation two. The human-written starting corpus is gone.

Gerstgrasser and colleagues asked whether that matches reality. On the actual internet, old data does not vanish when new data is published. Human-written text from 2015 is still there. Machine-generated text is added alongside it. The better model of the real situation is accumulation, not replacement.

They re-ran the experiments both ways, across language models, image models and other generative systems.

Under replacement, they reproduced the collapse. Under accumulation, where each generation trained on the original real data together with all synthetic data produced so far, test error did not grow without limit. It stayed bounded. Their theoretical analysis of a simplified model confirmed the same result: with accumulation, error has a finite ceiling regardless of the number of generations.

That finding substantially changes the practical message. Collapse is not an inevitable consequence of synthetic data existing in the world. It is a consequence of losing the real data.

How reassuring is that?

Not completely, for three reasons.

Bounded is not the same as unharmed. A model whose error stays below a ceiling may still be worse than it would have been with clean data. Dohmatob and colleagues studied this under the heading of strong model collapse. In their theoretical setting, even a very small proportion of synthetic data in the training mix, on the order of one per cent, could prevent performance from improving as the dataset grew. Adding more data did not wash the effect out. The result is for simplified models, and how far it carries to large systems is an open question, but it argues against complacency.

The proportion of real data falls over time. Accumulation keeps the old human-written data. It does not keep it at a constant share. If machine-generated content is added faster than human-written content, the real fraction shrinks, and the results on fresh data suggest that fraction matters.

Diversity erodes before quality does. Early collapse is the loss of tails. A model can remain fluent and accurate on average while becoming less varied and worse on unusual inputs. Studies comparing machine-generated text with human writing have found reduced lexical and syntactic variety. Standard quality metrics, which measure the average case, will not show this.

The fair summary is that catastrophic collapse requires conditions unlikely to hold at internet scale, while a slower narrowing remains a credible risk that is hard to see with ordinary metrics.

Is the web filling with synthetic text?

The risk depends on how much machine-generated content is entering the data that models train on. Direct measurement is difficult, but there are indications.

A study by Thompson and colleagues at Amazon examined multi-way parallel text on the web, meaning content that appears in many languages. They found that a large share of such content showed signs of machine translation, and that it was disproportionately low in quality and concentrated in lower-resource languages. For a language with little original web content, a substantial fraction of what exists online may already be machine output.

In a narrower domain, Liang and colleagues estimated the proportion of text in peer reviews at major AI conferences that had been substantially modified or produced by language models after ChatGPT’s release. Their estimates ranged from 6.5% to 16.9%.

These are specific measurements, not a census of the web. They show that the contamination is uneven. It is likely to be heaviest precisely where human-written data is scarcest, which is where a model has the least real signal to anchor it.

From a modelling problem to a provenance problem

If the key variable is the proportion and identity of real data in the mix, then the central capability is knowing where each piece of data came from.

For data an organisation generates itself, this is easy in principle and often neglected in practice. Synthetic examples should be labelled as such at creation, with the generating model and date recorded, and that label should travel with them. Many pipelines mix generated and collected data into one training file with no marker.

For data from outside, it is harder. Reliable detection of machine-generated text remains unsolved, and detectors perform worse on text by non-native writers and on edited output. Timestamps help: data from before late 2022 is very unlikely to be the output of a modern language model, which gives older archives a particular value. Source matters too. Data obtained directly from a known human process, such as a company’s own records, a commissioned annotation project or a licensed archive, has a provenance that scraped text does not.

This is one reason first-party and commissioned data is appreciating. Its value is no longer only that it is relevant. It is that its origin is known.

Should we worry less?

A sceptic could reasonably argue that the concern is overstated. Frontier developers filter their training data heavily. Much of the best recent progress involves deliberately training on synthetic data, including model-generated reasoning traces and distilled outputs, with good results. If synthetic data were as harmful as the collapse narrative implies, that would not work.

The observation is correct and consistent with the research. In those successful cases the synthetic data is curated: generated to a purpose, filtered by a verifier or a quality model, and combined with real data. The previous article described the conditions. Kazdan and colleagues, examining different training regimes, likewise found that outcomes depended heavily on how synthetic and real data were combined across generations.

The risk identified by the collapse literature is the uncurated case: synthetic data of unknown origin and quality entering training without anyone having chosen it. The remedy is curation and provenance, both of which are data-management disciplines.

What to do

Label synthetic data at source. Record that it is generated, by what, when, and from which inputs. Keep the label attached through every transformation.

Preserve the real data. Never overwrite or discard verified human-origin datasets in favour of generated ones. Version them and store them separately.

Track the ratio. For each training run, know the proportion of real to synthetic data, overall and by category.

Monitor the tails. Evaluate on rare cases and measure output diversity directly, since average quality metrics lag.

Prefer known sources. Where it matters, use data whose human origin can be established: first-party records, commissioned collection, dated archives, licensed corpora.

Take extra care in low-resource languages. The available web text is more likely to be machine-translated, and there is less human-written data to offset it.

What this changes