The Logarithm Problem

Why the tenth million examples teach a model far less than the first, and what that does to the case for “just collect more”.

In 2017, researchers at Google and Carnegie Mellon ran an experiment designed to answer a simple question. If you give a vision model 300 times more training data than ImageNet, how much better does it get?

They had the means to find out: an internal dataset called JFT-300M, with around 300 million images and noisy, automatically generated labels. The answer was that performance kept improving, which was the headline. The detail was in the shape of the curve. Performance on downstream vision tasks rose logarithmically with the volume of training data.

A logarithmic relationship is easy to state and easy to underestimate. It means each fixed gain in performance requires multiplying the data, not adding to it. Going from one million to ten million examples buys roughly what going from ten million to a hundred million buys. The second step costs ten times as much.

Both sides of the “more data versus better data” argument cite this paper. They are both right, and the reason is worth working through.

The case for volume

The volume argument has a distinguished history. In 2001, Banko and Brill showed that on a language task, simple algorithms trained on a billion words outperformed sophisticated ones trained on a million, and that the ranking of algorithms changed as data grew. In 2009, three Google researchers gave the idea its name in an essay called “The Unreasonable Effectiveness of Data”, arguing that for problems involving human behaviour, a lot of messy data beats a little clean data and an elegant model.

More recent evidence is stronger still. CLIP was trained on 400 million image–text pairs collected from the web with no manual labelling. ALIGN used 1.8 billion pairs of images and noisy alt-text, and its authors stated their conclusion plainly: scale can make up for noise. Rolnick and colleagues showed that deep networks can learn accurately even when correct labels are outnumbered many times over by randomly wrong ones, provided there are enough correct examples in absolute terms.

So the volume position is not naive. Under the right conditions, quantity does substitute for quality.

The conditions

There are two conditions, and they are the whole argument.

The first is that the noise is random. Rolnick’s result concerns labels corrupted uniformly at random. Random errors point in all directions and cancel as data grows. Systematic errors do not. If annotators consistently confuse two classes, or if a web-scraped caption source consistently describes one group of people in a particular way, more data reinforces the error. Work on real annotation noise bears this out: when Wei and colleagues re-labelled CIFAR with human annotators, they found that real-world label noise is structured by the content of the image and is harder for models to cope with than the synthetic noise most robustness methods are tested against.

The second is that data is nearly free. A logarithmic return is acceptable when the marginal example costs almost nothing, as with crawled web pages. It is ruinous when each example needs a radiologist, a native speaker of Maithili, or a robot arm in a real kitchen.

The long tail makes this sharper. A 2024 study by Udandarao and colleagues examined 34 multimodal models across five pretraining datasets and asked how well each model handled a concept as a function of how often that concept appeared in its training data. The relationship was log-linear: linear gains in performance required exponentially more examples of the concept. The models were not generalising to rare concepts so much as performing in proportion to exposure. For anything rare, which in most enterprise settings means most of what matters, undirected volume is the slowest possible route.

What selection buys

If volume gives logarithmic returns, the natural question is whether choosing examples carefully can do better. The research answer is a qualified yes.

Toneva and colleagues tracked how often each training example was learned and then forgotten during training. Many examples were never forgotten once learned, and a large share of those could be removed from the training set without hurting accuracy. Not all data is doing work.

Swayamdipta and colleagues took this further with what they called dataset cartography. By measuring a model’s confidence in each example across training, and how much that confidence varied, they sorted datasets into three regions. Easy examples helped optimisation but added little. Ambiguous examples, where the model wavered, contributed most to generalisation outside the training distribution. Hard examples, which the model rarely got right, were frequently labelling errors.

That taxonomy is practically useful. It says the examples most worth paying for are those the current model is uncertain about, and that the examples it consistently fails on deserve an audit before they deserve more training.

Sorscher and colleagues supplied the theory. Their 2022 analysis showed that with a good enough measure of example difficulty, error can in principle fall exponentially with the size of a pruned dataset, not as a power law. They were also candid about the gap between theory and practice: most existing pruning metrics performed poorly at ImageNet scale.

The same idea shows up in small language models. Microsoft’s phi-1, a 1.3-billion-parameter model, reached competitive results on Python coding benchmarks after training on a few billion tokens of filtered and synthetic “textbook quality” data, a fraction of what comparable models consumed. The authors’ explanation was the data.

Where the selection argument overreaches

Curation is not costless, and its advocates sometimes write as if it were.

Every selection method needs a model to do the selecting, and the choice of examples then carries that model’s blind spots. Data chosen as informative for one architecture is not always informative for the next. Pruning removes redundancy, but it can also remove the rare subgroups that happened to look redundant to the scoring function. And a curated dataset is a smaller dataset, which is a weaker defence against distribution shift nobody anticipated.

There is also an honest asymmetry between frontier labs and everyone else. For a lab pretraining a foundation model on trillions of tokens, heavy filtering has been important but so has sheer scale, and nobody has shown that a small curated corpus can replace broad pretraining. The selection argument is strongest downstream: in fine-tuning, in evaluation and in domain adaptation, where datasets are small enough for each example to be examined and expensive enough to justify it.

A way to decide

A useful test is to ask what the next thousand examples would cost and what they would teach.

If they are nearly free and drawn from the same distribution the model will face, collect them. If they are expensive, ask first which thousand. A model’s own uncertainty, its error analysis by subgroup, and its failures in production are all better guides to what to collect next than a target number.

A second test is to ask whom the model fails. Aggregate accuracy can keep rising while performance for a small but important group stays flat, because new data contains that group in the same small proportion as before. If the failures concentrate in a particular region, device, language or customer type, the remedy is data from that group, and no increase in general volume will supply it at a reasonable cost.

A third test is to ask what kind of errors the data contains. Random noise can be outvoted with redundancy or scale. Systematic noise needs a change in guidelines, annotators or sources, and no quantity of additional data will substitute for that.

What this changes