Whose Preferences?

Preference data has replaced labels as the scarce input for generative AI. Unlike a label, a preference always belongs to someone.

Ask ten people whether a photograph contains a dog and, barring odd cases, you will get one answer. Ask ten people which of two essays is better and you will get a vote.

That difference is easy to overlook, because the two kinds of data are stored the same way: an item, an identifier, a choice. They are collected through similar interfaces by similar workforces. But they are different kinds of object. The first is an attempt to measure a fact. The second is a record of what a particular person, in a particular situation, liked.

Generative AI runs increasingly on the second kind. As the previous article described, post-training methods learn from comparisons between outputs. Public leaderboards rank models by which one users preferred. Product teams tune on thumbs-up and thumbs-down. The question of whose preferences these are has moved from a footnote to a design decision.

What a preference is evidence of

A classification label has an implied referent outside the annotator. If two annotators disagree about whether an image shows a dog, one of them is probably wrong.

A preference has no such referent. If one rater prefers a concise answer and another prefers a thorough one, neither is mistaken. They want different things. Aggregating them produces an answer to a different question: what does this group, weighted this way, prefer on average?

Three properties follow, and each has consequences for how the data should be collected.

Preferences depend on the person. The PRISM dataset, built by Kirk and colleagues, collected the preferences of 1,500 participants from 75 countries across more than 8,000 live conversations with 21 language models, and linked each rating to the participant’s background. The authors found that which models were rated best depended on who was in the sample. Different subgroups, defined by demographics or geography, produced different rankings. There was no single ordering that a different choice of raters would have left unchanged.

Preferences depend on the context. The same person wants brevity when checking a fact and depth when learning a subject. A preference stripped of the task it was expressed in is hard to interpret.

Preferences can be mistaken about their own object. A rater may prefer the answer that sounds authoritative and be wrong about which answer is accurate. As the previous article noted, work by Hosking and colleagues found that assertiveness shifted human judgements of factual accuracy. This is where preference and fact become entangled: a rater’s preference for answer A is partly a belief that A is correct, and that belief can be false.

Where the preferences have come from so far

It is instructive to ask who has supplied the preference data that current systems learned from.

For early post-training work, the answer was small groups of contractors. InstructGPT used about forty, selected in part for agreement with the research team.

For public leaderboards, it is whoever turns up. Chatbot Arena, the most influential of these, lets visitors compare two anonymous models and vote. Its founding paper reported more than 240,000 votes. This is real usage by real people and far more varied than a contractor pool. It is also a self-selected population of people who visit model-comparison websites, asking the kinds of questions such people ask. A 2025 critique titled “The Leaderboard Illusion” raised further concerns, including that some developers were able to test many private model variants and publish only the best, and that access to arena data was unevenly distributed, which could let models be tuned to the arena’s particular distribution.

For product feedback, it is existing users, with all the selection that implies. People who dislike a product leave and stop providing feedback.

None of these is illegitimate. Each is a sample from some population, and none is a sample from “humanity” or from any particular enterprise’s customers.

Research on model outputs is consistent with this. Santurkar and colleagues compared the opinions expressed by language models with survey responses from different demographic groups in the United States and found that models aligned more closely with some groups than others, with the pattern shifting between model generations as training procedures changed. Durmus and colleagues at Anthropic did the same across countries using global survey questions and found that default responses were closer to the opinions of some populations, largely Western ones, than others.

The case for an average

The objection to all this is practical. A product ships one default behaviour. It has to be tuned to something. A broad average of reasonable people’s preferences is a sensible target, and worrying about whose preferences are represented is a luxury.

There is something to this. For many properties, such as coherence, relevance and following the instructions given, preferences converge across almost everyone. A response that ignores the question is bad for all raters.

But for a large class of properties, averaging produces a result nobody wanted. If half of users want formal language and half want casual, the mean satisfies neither. If preferences about directness differ between cultures, a model tuned to the pooled average is slightly wrong everywhere. And averages silently weight by headcount: a minority of users with distinct needs, whether speakers of a less common language or practitioners in a specialised field, contribute little to the mean and are served badly by it.

A second practical objection is that preference data is expensive, and specifying the rater population makes it more so. That is true. The alternative is to pay less for a signal whose meaning is unknown.

Three ways to handle plurality

Researchers have begun to describe what it would mean for a model to accommodate differing preferences, a programme sometimes called pluralistic alignment. Sorensen and colleagues distinguish several targets, which map onto practical choices.

Represent the spread. Where views differ on a contested question, the model presents the range instead of selecting one. This suits public-facing systems dealing with questions of value.

Make it steerable. The model can be directed toward a given style or standard through instructions or configuration. The preference data then needs to cover the range of styles, with each example tagged by which standard it reflects.

Match a defined population. The model’s default is tuned so that its behaviour matches the distribution of preferences in a specified group. This is what an enterprise usually wants: the model should prefer what our customers, or our underwriters, or our clinicians prefer.

There have also been experiments in sourcing the principles themselves from a deliberately chosen public. Anthropic and the Collective Intelligence Project ran a process in which roughly a thousand Americans contributed to and voted on principles for model behaviour, and a model was then trained against the resulting document. The exercise was small and exploratory, and it made the sampling question explicit.

Collecting preference data properly

For a team commissioning or building a preference dataset, the implications are concrete.

Define the population first. State whose preferences the model should reflect. Recruit raters to match, and record the composition of the pool.

Keep rater identity attached. Store a persistent, anonymised rater identifier and relevant attributes with every judgement. This allows analysis by subgroup, detection of individual raters with unusual patterns, and re-weighting later.

Separate fact from taste. Ask about correctness separately from style, and where correctness matters, have it assessed by someone equipped to check. A single “which is better” question lets a preference for confident prose masquerade as a judgement of accuracy.

Collect reasons. A short justification with each preference costs time and repays it. It shows what raters were attending to, exposes misreadings of the task, and can be used directly in training.

Report the disagreement. For each comparison, how split were the raters? High-agreement comparisons are safe to train on. Split ones need a decision about what the model should do when people differ.

Check the prompt distribution. Preferences are expressed on particular prompts. If the prompts do not resemble what the deployed model will be asked, the preferences will not transfer.

What this changes