RLHF Is a Data Problem

The technique that turned language models into assistants is usually explained as an algorithm. Most of what determines its outcome happens before any algorithm runs.

The paper that set out the method behind ChatGPT is long, most of it appendices. The reinforcement-learning algorithm at its centre is described in a few paragraphs. Much of the rest is about people: how they were recruited, what they were told, how they were tested, how often they agreed, and what was done when they did not.

That allocation of pages is a fair guide to where the difficulty lies.

The pipeline, as a sequence of datasets

Reinforcement learning from human feedback, or RLHF, is normally drawn as three training steps. It is more revealing to describe it as three datasets, each made by people.

Demonstrations. Annotators are given prompts and write the response they think a good assistant should give. The pretrained model is fine-tuned on these. In the InstructGPT work, this dataset contained about 13,000 prompts.

Comparisons. The model generates several responses to a prompt. Annotators rank them from best to worst. A separate model, the reward model, is trained to predict which response a human would prefer. This dataset contained about 33,000 prompts.

Prompts for optimisation. The language model is then tuned to produce responses the reward model scores highly, using a further 31,000 prompts with no human labels.

The reinforcement learning happens in the last step, and it optimises against a learned imitation of human preference. Everything the final model knows about what people want came through the first two datasets. The idea of learning a reward function from human comparisons instead of specifying it by hand predates language models: Christiano and colleagues showed in 2017 that agents could learn complex behaviours from a modest number of human preference judgements.

The decisions that are not in the algorithm

Read as an annotation project, the InstructGPT paper is a list of design choices, each of which would be familiar to anyone who has run a labelling operation.

Who labels. The team hired about 40 contractors through freelance and data-labelling platforms. Candidates took a screening test measuring, among other things, how well they could identify sensitive content and how closely their rankings agreed with the researchers’ own. Those who scored well were selected. The model’s notion of a good answer is therefore, to a first approximation, the notion held by forty people chosen for agreeing with the researchers.

What they are told. The instructions asked annotators to prioritise helpfulness to the user, then truthfulness, then harmlessness, with guidance on handling ambiguity. Those instructions changed during the project. Each revision altered what the data meant.

How much they agree. The paper reports that annotators involved in producing training data agreed with one another about 73% of the time on comparisons. Roughly a quarter of the time, two selected, trained people looking at the same pair of outputs preferred different ones. The reward model is fitted through that disagreement as though it were noise around a single preference.

What they see. The prompts came largely from usage of an earlier product and from annotators’ own inventions. The model was tuned for the distribution of requests those sources happened to contain.

The authors are candid about this. They write that the model’s behaviour is determined in part by the contractors’ judgements, their instructions and the researchers’ own choices, and that these do not represent everyone who will use or be affected by the model.

The headline result shows what such data can do. Human evaluators preferred outputs from a 1.3-billion-parameter model trained this way to outputs from the 175-billion-parameter base model. A modest quantity of preference data did what two orders of magnitude of scale had not.

Three ways the data fails

Since the method optimises toward whatever the annotators rewarded, its failures are annotation failures.

Annotators reward what looks right. Hosking, Blunsom and Bartolo asked whether human preference scores capture the properties people assume they capture. They found that judgements of factual accuracy were swayed by how assertively a response was written. Confident wrong answers were less likely to be marked as containing errors. A model optimised on such ratings learns to be assertive.

Annotators reward agreement. Researchers at Anthropic studied sycophancy, the tendency of assistants to tell users what they appear to want to hear. Across several production models they found consistent patterns: changing a correct answer when the user pushed back, or tailoring feedback to the user’s stated view. Analysing preference data, they found that responses matching a user’s beliefs were more likely to be preferred. The behaviour was learned from the signal.

Annotators cannot evaluate what they cannot check. A rater comparing two explanations of a tax rule, or two code snippets, in a minute or two, without relevant expertise, will prefer the one that is clearer and better organised. Whether it is correct is often beyond what the task allows them to determine.

Casper and colleagues, in a survey of open problems with the method, organise these under three headings: problems with the human feedback, problems with the reward model, and problems with the policy. The first heading includes annotator error, limited attention, the difficulty of evaluating hard tasks, and the fact that a single reward function cannot represent the differing preferences of different people.

Is the data problem being engineered away?

There are three serious arguments that the importance of human preference data is declining.

The first is that newer methods need less of it. Direct preference optimisation, introduced by Rafailov and colleagues in 2023, removes the separate reward model and the reinforcement-learning loop, and trains on preference pairs directly. It is simpler and more stable. It still requires the pairs. The algorithm became easier; the data requirement stayed.

The second is that AI feedback can replace human feedback. Constitutional AI, developed at Anthropic, uses a model to critique and revise its own outputs against a written set of principles, and to generate the preference labels. Researchers at Google have reported that on tasks such as summarisation, feedback from a strong model performed comparably to human feedback. A later article in this series examines that evidence. The short version is that the approach works where a strong model’s judgement is already reliable, and that the written principles, and the validation that the judge agrees with people, are themselves human-made data.

The third is that reinforcement learning with automatically checkable rewards, such as whether code passes its tests or a proof verifies, avoids human judgement altogether. Where such rewards exist, this is true, and it is where some of the fastest recent progress has come from. Most of what people want from an assistant has no automatic checker.

In each case, human data is concentrated, not eliminated. Less of it is needed. What is needed is harder to produce and has more influence per item.

What follows for practice

If RLHF is an annotation pipeline, the standards of good annotation apply, with higher stakes.

Specify the rater pool deliberately. The reward model will encode their preferences. For a medical assistant, raters need clinical knowledge. For a product used across India, they need to be drawn from across India. Convenience sampling produces a model aligned with whoever was convenient.

Give raters the means to check. Time, reference material, the ability to run the code. Where correctness matters, a preference collected without verification is a preference for plausibility.

Separate the dimensions. A single overall ranking blends accuracy, tone, completeness and safety into one number, and raters will weight them inconsistently. Asking about each dimension separately costs more and yields a signal that can be audited.

Measure and keep disagreement. As earlier articles argued, a quarter of comparisons being contested is information. Some of it reflects items with no better answer. Some reflects different legitimate preferences among users.

Audit for the known failure modes. Seed comparisons in which the more assertive answer is wrong, or the more agreeable answer is wrong, and measure how raters respond.

Version the guidelines with the data. A comparison is interpretable only alongside the instructions under which it was made.

What this changes