Evaluation Has Become Data Generation

Benchmarks used to be built once and used for a decade. They now leak, saturate and mislead within months, which makes evaluating a model a continuous exercise in making new data.

SWE-bench is one of the most watched benchmarks in AI. It tests whether a model can resolve real issues taken from open-source software projects, and it is graded automatically: the model’s code change either makes the project’s hidden tests pass or it does not. There are no subjective judgements and no human raters. It appears about as objective as an evaluation can be.

In 2024, OpenAI hired 93 professional software developers to read the benchmark’s problems. They reviewed 1,699 of them. For 38.3%, they judged the problem statement too underspecified for anyone to know what a correct solution would be. For 61.1%, they judged that the tests could reject valid solutions, for instance by requiring a function name the problem never mentioned. In total, 68.3% of the reviewed samples were filtered out.

From what remained, the team assembled a subset of 500 problems that people had confirmed were solvable and fairly graded. On the original benchmark, the best configuration of GPT-4o at the time scored 16%. On the verified subset, it scored 33.2%.

The model had not changed. Its measured ability doubled because the measurement was repaired. A benchmark with no humans in it turned out to need ninety-three of them.

That episode contains most of what has changed about evaluation.

Three ways benchmarks stop working

They contain errors. An earlier article in this series described label errors in classic test sets. SWE-bench shows the same problem in a modern agentic benchmark: tasks that are ambiguous, graders that are wrong. An automatic grader is a labelled dataset. Somebody wrote the tests, and they can be mistaken.

They leak. Language models are trained on very large crawls of the public web. Benchmarks are published on the public web. A model may have seen the test questions, and their answers, during training.

The evidence that this affects scores is direct. Researchers at Scale AI wrote more than a thousand new arithmetic word problems matched in style and difficulty to GSM8K, a popular benchmark, and compared model performance on old and new. Several model families scored measurably worse on the fresh problems, with drops of up to 8%, and the size of the drop correlated with signs that a model had memorised the original set. The strongest models showed little decline, which is itself informative: contamination inflates some scores and not others, so it scrambles rankings.

A study from Apple took a different approach, generating variants of the same benchmark’s problems with only names and numbers changed. Performance varied across variants of what was logically the same question, and fell when an irrelevant clause was added. Oren and colleagues developed a statistical test that can detect whether a model has memorised a benchmark, by checking whether it assigns higher probability to the test examples in their published order than in shuffled order.

They saturate. When top models all score above 90%, the remaining differences are within the noise of the benchmark’s own errors, and it can no longer rank them. The useful lifetime of a new benchmark has shortened as models have improved faster.

Under all three sits a problem of validity. Raji and colleagues, in a paper titled “AI and the Everything in the Whole Wide World Benchmark”, argued that general-purpose benchmarks are taken as measures of broad capability that they were never designed to measure. A high score certifies performance on that test, under those conditions.

The gap between benchmark and deployment

For an enterprise, the more pressing issue is that even a perfect public benchmark measures the wrong thing.

A model’s score on general coding tasks says little about how it handles one company’s codebase, with its conventions and its legacy systems. A model’s score on medical licensing questions says little about how it summarises discharge notes in a particular hospital’s format. Recht and colleagues’ finding that accuracy fell by 11 to 14 points when ImageNet models met a carefully reconstructed version of their own test set shows how sensitive results are to small changes in the data, and deployment involves much larger ones.

Agentic systems widen the gap further. Kapoor and colleagues, in a study titled “AI Agents That Matter”, found that agent benchmarks often lacked proper held-out sets, that reported results were difficult to reproduce, and that leaderboards ignored cost, so a system could top the table by making many expensive attempts. Evaluating an agent means evaluating a sequence of actions in an environment, and each test case requires someone to build the environment, define success and confirm that success is achievable.

Anthropic’s own account of the difficulties is direct. In a 2023 essay on the challenges of evaluating AI systems, its researchers described how multiple-choice benchmarks were sensitive to trivial formatting changes, how third-party frameworks required substantial effort to run faithfully, and how human evaluations were slow and hard to make consistent. The conclusion was that robust evaluation is much harder than it looks from outside.

Evaluation as a production process

If static benchmarks decay, evaluation becomes something that has to be manufactured continuously. Several approaches from the research literature describe what that looks like.

Dynamic benchmarks. Dynabench, from Kiela and colleagues, replaced the fixed test set with a loop. People write examples intended to fool the current best model. Examples that succeed, and that other people verify, go into the next round’s dataset. The benchmark moves as models improve.

Behavioural testing. CheckList, from Ribeiro and colleagues, borrowed from software testing. Instead of one aggregate score, a model is tested against a matrix of specific capabilities, with test cases written or templated by people: does it handle negation, does its answer survive a change of name, does it stay consistent under paraphrase. The authors found failures in commercial systems that had passed standard benchmarks.

Private and refreshed test sets. The simplest defence against contamination is not to publish. A test set that has never been on the internet cannot be in the training data. This requires that the set be created originally, kept under access control, and replaced periodically, because every query to a hosted model is also a potential leak. Jacovi and colleagues proposed practical measures for public benchmarks as well, including encrypting test data and refusing to send it unprotected to model APIs.

Human verification of automatic graders. The SWE-bench exercise is the template. Before trusting an automated metric, have qualified people confirm that the tasks are well posed and the grading is fair.

Each of these is a data-production activity. Each requires people with relevant skill writing, checking and curating examples to a specification. Evaluation data has become a product with a manufacturing process, a shelf life and a security requirement.

Why not rely on public benchmarks and model judges?

There is a reasonable case for a lighter approach. Public benchmarks are free and comparable. Automated judges, discussed in the previous article, are fast. A team could select a model from leaderboard results and monitor it with model-graded checks, without building any evaluation data of its own.

For low-stakes uses, that may be sufficient. Its limits are those already described. Leaderboard position reflects performance on tasks that may be contaminated and are not yours. Model judges need calibrating against human judgement on your task. And neither can tell you about failure modes specific to your data, which are the ones that will cost you.

A second objection is that custom evaluation is expensive. In absolute terms it is modest. A few hundred carefully built, expert-verified test cases is a small dataset. Compared with the cost of selecting the wrong model, or shipping a regression nobody measured, it is among the better investments available.

What an evaluation programme contains

A serious evaluation capability has five parts.

A private core set. Several hundred to a few thousand cases drawn from real or realistic inputs, with answers or rubrics established by qualified people. Never published. Access logged.

A slice structure. Cases tagged by the dimensions that matter: task type, difficulty, language, customer segment, risk level. Results are reported by slice.

A refresh cycle. A portion of the set retired and replaced on a schedule, with new cases drawn from recent production data and newly observed failures.

A failure pipeline. Every significant production error becomes a test case. This is how the evaluation set learns.

A verification step. People periodically confirm that test cases are still well posed and that automated graders still agree with expert judgement.

What this changes