When AI Grades AI

Language models now generate much of the feedback used to train and evaluate other language models. The evidence on when that works is more specific than either its advocates or its critics suggest.

A team building a customer-support assistant needs to know whether this week’s version is better than last week’s. There are four thousand test conversations. Having people read each response and score it would take days and cost more than the team’s evaluation budget for the quarter.

So they do what almost every team now does. They write a prompt describing what a good response looks like, give it to a strong language model along with each conversation, and ask it for a score. The whole run finishes before lunch.

The practice is called LLM-as-a-judge, and it has become the default way of evaluating generative systems. Its close relative, reinforcement learning from AI feedback or RLAIF, uses the same idea in training: a model supplies the preference labels that human raters used to supply.

Both raise an obvious question. If a model is doing the grading, what is the grade worth?

The evidence that it works

The affirmative case rests on several careful studies.

Zheng and colleagues, in the paper that introduced the MT-Bench benchmark, compared GPT-4’s preferences between pairs of chatbot responses with those of human experts. Agreement between the model judge and humans exceeded 80%, which was the same level at which the humans agreed with each other. By that measure the model was as good a judge as another person.

On the training side, researchers at Google compared reinforcement learning from AI feedback with the human version on summarisation and dialogue tasks. Policies trained on preferences generated by a language model were rated by humans as comparable to policies trained on human preferences. They reported a similar result even when the judge model was the same size as the model being trained.

Anthropic’s Constitutional AI took the idea furthest. A written list of principles, the constitution, was given to a model, which then critiqued and revised its own responses and chose between alternatives according to those principles. The resulting preferences were used for training with no human labels on harmfulness. The model produced was both less harmful and, by the authors’ measures, no less helpful than one trained with human feedback.

These are substantial results. For a meaningful set of tasks, a strong model’s judgement tracks human judgement well enough to train and evaluate on.

The evidence on how it fails

The same literature documents consistent ways in which model judges go wrong, and they are not random.

Position. Zheng and colleagues found that judges tended to favour whichever response was presented first. Wang and colleagues showed the effect was large enough to be exploited: by changing the order in which two responses appeared, they could make either one win a majority of comparisons.

Verbosity. Longer responses were preferred even when the additional length added nothing.

Self-preference. Judges rated outputs from themselves, or from closely related models, more favourably. Panickssery and colleagues found that language models could recognise their own outputs at better than chance, and that the strength of that recognition correlated with the strength of the bias. An evaluation in which a model family judges itself against competitors is not neutral.

Limited competence. A judge cannot reliably grade what it cannot do. Zheng’s study found judges struggled on mathematics and reasoning questions when they could not solve the problem themselves, and were sometimes misled by a wrong answer placed in front of them.

Uneven reliability across tasks. A large study by Bavaresco and colleagues tested model judges against human judgements across twenty language-evaluation datasets. Agreement varied widely by task and by the property being judged. Their conclusion was that model judges should be validated against human judgements for each task before being relied on.

There is also a structural problem with using AI feedback for training. If the judge shares systematic blind spots with the model being trained, which is likely when they come from the same family or the same training data, those blind spots are reinforced. Human raters, whatever their limits, fail in different ways from the model.

Criteria do not write themselves

A less discussed difficulty concerns the judge’s instructions.

Shankar and colleagues studied how practitioners build model-based evaluators and observed a phenomenon they called criteria drift. People did not know their evaluation criteria in advance. They discovered and revised them while looking at outputs. Seeing a response they disliked for a reason they had not anticipated, they added a criterion. Seeing the judge apply a criterion too literally, they reworded it.

That means the judge’s prompt is a living document, written by people, encoding their evolving sense of quality. It plays the role an annotation guideline plays for human raters, and it has the same failure modes: ambiguity, omitted cases and unstated assumptions. A judge is only as good as its rubric, and writing the rubric is the human-in-the-loop work that automation does not remove.

Constitutional AI makes the same point at a larger scale. The model generates its own feedback, but from a constitution that people wrote. The human contribution was compressed from tens of thousands of comparisons into a page of principles. It was not eliminated, and each word on that page has more influence than any individual comparison had.

A division of labour

The honest reading of the evidence is that neither “AI can replace human feedback” nor “AI cannot grade AI” is right. A more useful statement has three parts.

Model judges are good at scale and consistency. They apply a rubric to ten thousand items without fatigue, at low cost, with reproducible results. For regression testing, where the question is whether a change made things better or worse on a fixed set, that is most of what is needed.

Humans are needed for calibration. A sample of the judge’s decisions has to be compared with decisions by qualified people, per task and per criterion. If agreement is high, the judge can be trusted on that task. If not, the rubric or the judge needs work. This calibration set is a dataset in its own right, to be built with the care this series has argued for throughout: appropriate raters, measured agreement, recorded ambiguity.

Humans are needed where the judge is out of its depth. Specialised domains, novel failure types, questions where correctness requires checking a source. Here the judge’s score should be treated as a filter that selects cases for expert review, not as a verdict.

The practical design is a pyramid. A model judge scores everything. People audit a stratified sample, weighted toward low-confidence and high-stakes cases. Experts adjudicate the disagreements. Results from the upper levels are used to revise the rubric and recalibrate the judge.

Making a judge trustworthy

Several inexpensive controls address the documented biases.

Randomise order and average. Present each pair in both orders. If the verdict flips, record a tie.

Use a different model family as judge. Do not have a model grade its own outputs or those of its siblings when comparing against others.

Ask for a rubric-based score, not an overall impression. Specific criteria with defined levels are applied more consistently than “rate from one to ten”.

Require reasoning before the score. Having the judge explain its assessment first often improves agreement with humans and gives auditors something to read.

Give the judge the reference. For factual tasks, supply the correct answer or the source document. A judge asked to verify against evidence is doing an easier and more reliable task than one asked to know the answer.

Use a panel. Several judges from different families, with disagreements escalated to people, reduce the effect of any single model’s quirks.

Re-calibrate when anything changes. A new judge model, a revised rubric, or a shift in the kind of outputs being judged each invalidate the previous calibration.

What automated feedback cannot do

Two limits deserve stating plainly.

A model judge cannot tell you what you should want. Whether a support assistant should be brief or thorough, cautious or direct, is a decision about the product and its users. As the previous article argued, that is a question about whose preferences count.

And a model judge cannot detect a failure nobody thought to describe. Rubrics list known problems. New ones are found by people reading outputs with open attention, which is slow, unscalable and the origin of every criterion a judge is later asked to apply.

What this changes