Red Teaming Is Dataset Construction

Adversarial testing of AI systems is usually described as a security exercise. Its lasting output is a dataset, and it inherits every question this collection has asked about datasets.

In 2023, researchers at Brown University took a set of requests that GPT-4 reliably refused, such as instructions for causing harm, and ran them through a translation service into languages with little presence on the internet, including Zulu and Scots Gaelic. They submitted the translated requests and translated the responses back into English.

The model’s safeguards, which had held in English, largely gave way. Across the low-resource languages tested, the combined attack succeeded 79% of the time on a standard benchmark of harmful requests. In high-resource languages the success rate was far lower.

No sophisticated technique was involved. The weakness existed because the data used to teach the model what to refuse was overwhelmingly in English and a few other major languages. The safety training had a coverage gap, and the gap had the same shape as the data.

That is a finding about red teaming as much as about the model. Somebody had to think of trying it. Whether a vulnerability is found depends on who is looking and what they know to look for.

What red teaming produces

The term comes from military and security practice, where a designated team plays the adversary. Applied to AI, it means deliberately trying to make a system behave badly: produce harmful content, leak private information, follow malicious instructions, give dangerous advice.

The activity gets the attention. The output is what lasts. A red-team exercise produces a collection of inputs that caused failures, the outputs they caused, and a classification of what went wrong. That is a dataset. It has three downstream uses, all of them data uses.

As evaluation data. The successful attacks become a test suite. Every future version of the model can be checked against them.

As training data. The attacks, paired with the responses the model should have given, are used to teach refusal or safer behaviour.

As classifier data. Examples of harmful inputs and outputs train the separate filters that screen traffic in production.

Anthropic’s early work on the subject made this explicit. Ganguli and colleagues ran a large human red-teaming effort against language models of several sizes and released the result as a dataset of 38,961 attacks, with analysis of what kinds of harmful output were elicited. The paper is as much about how to collect and label such data as about the vulnerabilities found. Among its observations was that models trained with human feedback became harder to red team as they grew larger, while other configurations did not.

The annotation problems, again

Once red teaming is seen as data production, the questions from the earlier parts of this series apply directly.

Coverage. A red-team dataset represents the attacks its creators thought of. The translation attack shows what follows when the team’s languages are narrower than the users’. Deng and colleagues documented the same pattern across a wider set of languages: unsafe responses became more frequent as the language’s resource level fell. Coverage gaps in safety data map onto gaps in who was recruited to probe.

Whose judgement. Whether an output is harmful is frequently contested. A frank answer about medication doses is a safety failure in one context and appropriate in another. Weidinger and colleagues at Google DeepMind argue that safety evaluation has to consider not only what a model can output but who is interacting with it and in what setting. Red-team labels are preference data with the stakes raised, and the problems of rater selection discussed in previous articles recur.

Agreement. Two red-teamers may disagree about whether an attack succeeded, how severe the result was, and which category it belongs to. Without a taxonomy and measured agreement, counts of “successful attacks” are not comparable between exercises.

Documentation. A finding that is not written down in reproducible form, with the exact input, the model version, the settings and the output, cannot be re-tested and cannot be used.

Feffer and colleagues surveyed published red-teaming exercises and found wide variation on all of these: who took part, what they were asked to do, what resources they had, what counted as a finding and what was reported afterwards. Their title asked whether red teaming, as practised, was a silver bullet or security theatre. Their answer was that without agreed structure it could not serve as the assurance mechanism policy discussions were treating it as.

Scale and structure

There are broadly three ways to produce adversarial data, and they complement each other.

Expert red teaming uses small groups with deep knowledge of a domain, such as biosecurity, child safety, fraud or a particular culture and language. It finds the failures that require knowledge to imagine. It is slow, expensive and does not provide broad coverage.

Crowd red teaming uses many people with varied backgrounds. The Generative Red Team event at the DEF CON security conference in 2023 brought more than two thousand participants to test models from several developers. Breadth of perspective is the benefit. Consistency of method and labelling is the cost.

Automated red teaming uses models and algorithms to generate attacks. Perez and colleagues showed that language models could write evaluation datasets testing for a wide range of behaviours, with human reviewers confirming the quality of most generated examples. Zou and colleagues showed that an automated search could discover strings of characters that, appended to a request, caused several different models to comply with harmful instructions, and that these strings transferred between models. Benchmarks such as HarmBench standardise the comparison of automated methods.

Automation gives volume and repeatability. It explores variations on what it was pointed at. The translation attack, and others like it, came from a person having an idea about how the system might be mismatched to the world. The usual pattern is that humans discover a new class of failure and automation then fills in its many instances.

What regulation now expects

Adversarial testing has moved from voluntary practice toward obligation. The European Union’s AI Act requires providers of general-purpose models with systemic risk to perform and document adversarial testing. The United States National Institute of Standards and Technology published a profile of its AI Risk Management Framework for generative AI in 2024, which lists red teaming among its recommended measures.

These frameworks ask for documentation: what was tested, by whom, with what result. That is a requirement for data with provenance.

The limits of the exercise

Three cautions are in order.

Red teaming demonstrates the presence of failures and cannot demonstrate their absence. An exercise that finds nothing may mean the system is robust or that the team lacked imagination, time or the right languages.

Training on red-team data can produce a model that resists the specific attacks it was shown and little else. If the same findings are used for both training and evaluation, the evaluation will overstate robustness. Held-out adversarial data, kept from training, is necessary for an honest measurement, for the same reasons the previous article gave for any test set.

And the work has a human cost. Red-teamers and the annotators who label harmful content are exposed to disturbing material as a condition of the job. Research on content moderators has documented psychological harm, and reporting on data workers has raised the same concern for those labelling toxic content for AI systems. An operation that produces safety data has a duty of care to the people producing it, including limits on exposure, support and fair pay.

Running it as a data programme

Treated as dataset construction, a red-team programme has a specification.

A taxonomy of harms, tailored to the application, with definitions and severity levels. Without it, findings cannot be counted or compared.

A recruitment plan that deliberately covers the languages, regions, domains and user groups the system will meet. Diversity here is a coverage requirement.

A recording standard: the exact input, the model version and configuration, the full output, the tester’s classification, and a second person’s verification.

A split between findings used for training and findings held back for evaluation.

A refresh schedule. Models change, attackers adapt and yesterday’s suite goes stale.

A welfare policy for the people doing the work.

What this changes