The Synthetic Data Paradox

Generated data works best in the domains we already understand well enough to simulate. It works worst where real data is hardest to get, which is where it is most often proposed.

In 2024, a system from Google DeepMind solved geometry problems from the International Mathematical Olympiad at a level approaching that of a gold medallist. It had been trained on one hundred million examples of geometric theorems and proofs. Not one of them was written by a person. All were generated by a program that built random geometric figures and derived what could be proved about them.

Two years earlier, a group of researchers had examined a different use of synthetic data. Organisations wanted to share sensitive records, such as medical data, without exposing individuals. The proposal was to train a generative model on the real records and release its synthetic output instead. Stadler and colleagues tested whether this protected privacy. Their conclusion, published at a leading security conference, was that it offered no better trade-off than older anonymisation techniques. Synthetic data that preserved enough detail to be useful also preserved enough to leak information about real individuals, especially unusual ones. Synthetic data that was safe was of little use.

One result is a triumph of synthetic data. The other is a warning about it. The difference between them is the whole subject.

What a generator knows

Every synthetic dataset is produced by something: a physics simulator, a rendering engine, a rule-based program or a trained generative model. The data it emits can contain no more information about the world than the generator has.

That gives a simple way to sort the uses.

When the generator encodes rules that are known to be true, synthetic data is close to ideal. Geometry has axioms. A program that applies them correctly produces valid theorems, unlimited in number and perfectly labelled. The same holds for other formal domains: arithmetic, code that can be executed and tested, games with fixed rules. The generator knows the domain better than any dataset could.

When the generator is a faithful simulator of physical processes, synthetic data is useful to the degree the simulation is accurate. Renderers can produce images with exact labels for depth, object boundaries and position, which are expensive or impossible to obtain from photographs. A Microsoft team trained face-analysis models entirely on rendered synthetic faces and achieved accuracy competitive with models trained on real photographs, having invested heavily in the realism of the rendering. Robotics researchers train policies in simulation and transfer them to hardware, using a technique called domain randomisation, which varies textures, lighting and physics so widely that the real world looks like one more variation. The limiting factor is the remaining difference between simulation and reality, which practitioners call the sim-to-real gap.

When the generator is a model trained on real data, the synthetic data is a resampling of what that model learned. It can be valuable. It cannot contain anything the model did not absorb, and it will under-represent what the model absorbed weakly: the rare cases.

That third category is where most enterprise proposals fall, and where the paradox applies.

Where learned generators help

The evidence that model-generated data can improve models is solid within limits.

Azizi and colleagues at Google fine-tuned an image-generation model and used it to produce additional training images for ImageNet classes. Adding these to the real training set improved classification accuracy. The synthetic images were an augmentation of the real data, not a substitute for it.

Language models are routinely used to produce training data for smaller models. Microsoft’s phi models were trained in part on synthetic “textbook-like” text and exercises generated by a larger model, and reached strong performance for their size. This is a form of distillation: knowledge held by a large model is transferred to a small one through generated examples.

Other dependable uses include rebalancing, by generating more examples of an under-represented class the generator does know about; creating variations for robustness, such as paraphrases or changes of format; and producing test cases to a specification.

What these have in common is that the generator already possesses the relevant knowledge. The synthetic data moves it, reshapes it or multiplies it.

Where they do not

The long tail. A learned generator reproduces the distribution it was trained on, and tends to do so with reduced variety. Rare events, unusual presentations and minority subgroups are thin in the training data and thinner still in the output. A study by He and colleagues on using synthetic images for recognition found benefits concentrated in settings with little or no real data, with gains diminishing as real data became available, and noted the gap in diversity between generated and real images. If the reason for wanting synthetic data is that the real data lacks edge cases, a generator trained on that real data lacks them too.

New domains. An enterprise that wants a model for its own documents, in its own terminology, cannot get there by asking a general model to imagine such documents. The general model will produce plausible text that resembles what it has seen elsewhere. The specifics that make the enterprise’s data distinctive are exactly what it does not know.

Privacy. As Stadler’s work showed, synthetic data is not automatically anonymous. Generative models can memorise and reproduce training examples. Carlini and colleagues extracted recognisable copies of training images from diffusion models, with duplicated and unusual images most at risk. A synthetic dataset derived from personal records needs the same privacy analysis as the records.

Errors at scale. A human annotator who misunderstands a guideline mislabels some items. A generator with a systematic flaw embeds it in every example, with perfect consistency. Because synthetic data is cheap, the flawed dataset can be very large.

The verification problem

There is a second half to the paradox. To know whether a synthetic dataset is good, you have to compare it with reality. That requires real data.

The Royal Society and the Alan Turing Institute, in a 2022 overview of the field, made the point that the utility and privacy of synthetic data cannot be assumed from the method that produced it. They have to be evaluated empirically for each use. Evaluating utility means training a model on the synthetic data and testing it on real data. Evaluating fidelity means comparing distributions against a real sample.

So synthetic data does not remove the need for real, well-labelled data. It changes the quantity needed and what it is used for. A small, trusted real dataset becomes the reference against which large volumes of generated data are validated. If that reference is poor or absent, nobody can tell whether the synthetic data is helping.

A sensible forecast for most applied settings is a ratio that shifts, not a replacement. More of the training volume becomes synthetic. The real data that remains is concentrated in evaluation, in seeding the generator, and in the rare cases the generator cannot produce.

The case for optimism

The sceptical reading should not be overdone. There are three reasons to expect synthetic data to matter more.

Generators are improving. Each increase in the fidelity of simulators and generative models extends the range of problems where their output is good enough.

The public data supply is tightening, as the first article in this series described. Where real data is legally restricted, exhausted or prohibitively expensive, an imperfect synthetic alternative may be the only option.

And the most striking recent progress has come from systems that generate their own training signal in domains with a verifier: programs that can be run, proofs that can be checked, games that can be scored. Where an automatic check exists, a model can generate attempts, keep the ones that pass, and learn from them, improving beyond its starting point. This is the regime of the geometry result.

The critical word is verifier. In these cases new information does enter the system, from the checker. Without an external check, a model learning from its own output has no source of correction, a problem the next article examines.

A decision procedure

For a team considering synthetic data, four questions settle most cases.

What is the source of truth in the generator? Known rules, a validated simulator, or a model trained on data. The answer bounds what the synthetic data can contain.

What is missing from the real data, and does the generator have it? If the gap is volume of well-understood cases, generation helps. If the gap is rare or novel cases, it probably does not.

How will it be validated? Name the real dataset that will serve as the test. If none exists, build that first.

What is the mix? Plan to combine synthetic with real data, and measure how performance changes as the proportion varies.

What this changes