Short answer. Three decisions constrain a red-teaming dataset before any prompt is written: whose taxonomy you adopt, whether you build, borrow or collect, and how you split automated against human testing. The taxonomy matters most, because it determines what can be found at all — whatever sits outside it is invisible rather than absent, and CHI 2026 research found the construction approach itself shapes how practitioners conceive of risk. This is now a compliance question too: the EU AI Act reached full enforcement in August 2026, with Article 55 requiring documented adversarial testing for GPAI models carrying systemic risk.
Key takeaways
- CHI 2026 research found that the approach taken to dataset creation shapes practitioners' conceptualisation of risk, and the taxonomy determines what can be found — whatever falls outside it is invisible rather than absent.
- The EU AI Act reached full enforcement in August 2026. Article 55 requires documented adversarial testing for GPAI models with systemic risk, Article 9 mandates lifecycle testing procedures, and penalties reach €35 million or 7% of global turnover.
- Public adversarial datasets have become less reliable because models are increasingly trained to pass them; a benchmark inside the training distribution measures memorisation, not safety.
- Automated frameworks reportedly find vulnerabilities at 3.9 times the rate of manual testing, but automation provides breadth while human red teamers provide depth and novel attack discovery.
- Brown University research found translating unsafe prompts into low-resource languages produced harmful responses roughly 80% of the time, and evaluation across 79 languages found unsafe response rates rising by as much as 25 percentage points relative to English.
What does building a red-teaming dataset actually require?
Red teaming is the practice of deliberately probing a model with adversarial prompts to surface unsafe or policy-violating behaviour before real users find it. Building a dataset for it comes down to three decisions made before any prompt is written: whose taxonomy of harm to adopt, whether to build, borrow or collect the adversarial material, and how to split effort between automated tooling and human testers.
There is a finding in a 2026 CHI paper on red teaming practice that reframes the entire exercise. Researchers interviewed practitioners about how they build adversarial datasets and found that the approach taken to dataset creation shaped their conceptualisation of risk. Teams building on existing datasets inherited the definitions of harm already embedded in the source material; teams building from scratch selected their own categories.
The taxonomy is not a neutral container for findings. It determines what can be found. A taxonomy is the documented set of harm categories a red-teaming programme tests against, and a programme with fourteen categories will discover harms in fourteen categories — whatever falls outside them is not a negative result, it is invisible.
A note on scope: this piece addresses how safety datasets are built, staffed and governed as a data operation. It is not a catalogue of attack techniques, and it stays at the level of methodology rather than method. Organisations doing this work well publish their processes, not their payloads.
Why is red teaming now a compliance requirement, not just a research exercise?
The regulatory position hardened considerably in 2026, and adversarial testing is no longer optional for models above a risk threshold.
The EU AI Act reached full enforcement in August 2026. Article 55 requires documented adversarial testing for general-purpose AI models with systemic risk. Article 9 mandates risk management systems including testing procedures throughout the lifecycle. Non-compliance carries penalties up to €35 million or 7% of global annual turnover.
In the United States, the NIST AI Risk Management Framework MEASURE function formalises adversarial testing as standard practice, and federal procurement guidance increasingly references red teaming as a requirement. The question for most organisations has moved from whether to do this to how to do it in a way that produces defensible documentation — meaning the dataset, the taxonomy, the coverage, the methodology and the results, not a summary paragraph saying testing occurred.
What three decisions determine everything downstream?
Three decisions, made early, constrain everything that follows: whose taxonomy to adopt, whether to build, borrow or collect the adversarial data, and how to balance automated tooling against human testers.
The CHI research identifies three critical moments in this work: defining and framing the task, developing the adversarial dataset, and evaluating models against it. Each carries a decision that constrains everything after it.
Reusing an established taxonomy gives methodological consistency and comparability with published results. It also means inheriting someone else's definition of harm, built for someone else's deployment context. The alternative — building categories from your own risk assessment — produces more relevant coverage and makes results incomparable to anyone else's. Neither is wrong; what is wrong is making the choice by default, which is what happens when a team downloads a public dataset and starts running it without examining what it does and does not classify as harmful. Analysis of one widely used safety dataset noted that responses demonstrating human-like emotion or behaviour were labelled as safe, leaving emotional manipulation risks entirely outside the measurement — not an annotation error, but a category that was never drawn.
On build, borrow or collect: practitioners described three sources with a clear hierarchy of perceived value. Reusing well-known datasets was seen as a way to ensure methodological consistency; building from scratch, or collecting real human-LLM interactions, was perceived as producing unique and more valuable data. The hard operational reason that perception is correct is that public adversarial datasets have become less reliable because models are increasingly trained to pass them. A benchmark is a fixed, published test set, and once one enters a model's training distribution it stops measuring safety and starts measuring memorisation. A public dataset tells you your model handles known attacks, and nothing about novel ones — which are the entire point.
On automated versus human testing: automated red teaming frameworks reportedly discover vulnerabilities at 3.9 times the rate of manual expert testing while covering multiple threat categories simultaneously, a real efficiency argument. But the framing that holds up across sources is that automated tools provide breadth and human red teamers provide depth, novel attack discovery and domain expertise. Anthropic's Policy Vulnerability Testing and OpenAI's external red teaming network are structured differently but rest on the same premise: domain-expert humans construct probes no automated attacker would think to write.
What do frontier AI labs publish about their red-teaming process?
Frontier labs publish red-teamer counts, country and language coverage, and named methodology documents, which sets a reference standard for what serious coverage looks like.
The GPT-4o System Card reports over 100 external red teamers across 29 countries speaking 45 different languages. The Operator System Card, covering OpenAI's web-browsing agent, reports 20 countries and 24 languages for that surface specifically. OpenAI's methodology is published in "Approach to External Red Teaming for AI Models and Systems" by Ahmad and colleagues. Anthropic's Policy Vulnerability Testing describes in-depth qualitative testing with external subject-matter experts on specific usage-policy topics.
Two things stand out: the scale of external involvement — this is not an internal team running a checklist — and the fact that language and country coverage is reported as a headline metric. Forty-five languages is a deliberate statement that safety behaviour varies by language and that testing in one language does not generalise. Most enterprise red teaming programmes test in English only, then report a safety result as though it applied to the entire deployment.
Why is the multilingual gap the largest weakness in most programmes?
Safety alignment is trained per language, so a model can refuse a request in English and comply with the same request in a language its guardrails were never taught.
This deserves attention because the evidence is unusually clear and the implications are underappreciated. A guardrail is a trained behaviour that refuses or redirects unsafe requests, and it exists only in the languages it was taught in. Researchers at Brown University demonstrated this by taking unsafe English prompts, translating them into low-resource languages such as Zulu using freely available translation, and sending them to GPT-4. The approach produced harmful responses roughly 80% of the time, against a model that refused the same requests in English.
The picture has evolved rather than resolved. A 2026 study testing African languages including Kiswahili, isiXhosa and isiZulu found that straightforward translation attacks no longer succeed as easily — genuine progress — but that conversations spread across multiple turns still succeed at high rates. Separate evaluation across 79 languages found unsafe response rates rising by as much as 25 percentage points as prompts moved from English into low-resource languages.
A guardrail that fails in any language is a guardrail anyone can route around using a free translation tool, which makes multilingual safety data a security control rather than a diversity initiative. The field is responding with language-specific safety datasets, including work on Albanian, but coverage remains thin relative to the number of languages models are deployed in. Programmes doing multilingual data collection at enterprise scale treat language coverage as a testing requirement, not an afterthought.
What does a usable red-teaming dataset actually contain?
A usable dataset pairs every adversarial prompt with a labelled model response, a harm category, a severity grade and an outcome judgement — a list of prompts alone is not a dataset.
The 2022 Anthropic red team release illustrates the gap: 38,961 red team attacks spanning twenty categories, substantial scale, but analysis noted that the absence of labelled responses reduced its effective utilisation for both automated red teaming and evaluation. A dataset that supports both evaluation and training needs, per record: the prompt or conversation, including the full multi-turn sequence where relevant, since multi-turn is where much of the current risk sits; the model response, verbatim; a harm category label from a documented taxonomy, with the taxonomy version recorded; a severity grading, because "unsafe" is not one thing and remediation priority depends on degree; a success or failure judgement on whether the attempt achieved its objective; attack technique classification, tracked so that technique diversity can be measured rather than assumed; language and locale; and annotator identity and adjudication history, for the same reasons any annotation programme needs a documented review chain.
One practitioner organisation notes that it trains red teamers across categories of adversarial technique and systematically tracks technique diversity to ensure comprehensive coverage across engagements — a discipline worth copying, since without it a team drifts toward the techniques individuals find most productive and coverage silently narrows. On labelling philosophy, the guidance from practitioners is consistent: conservative annotation, where safety labels err toward caution, produces the most reliable training data for defensive systems. A false positive costs a refused benign request; a false negative costs the thing the programme exists to prevent.
How do red-teaming programmes quietly lose coverage?
Coverage narrows through three failure patterns that all look like success on a dashboard: technique concentration, category imbalance, and benchmark contamination.
Individual red teamers develop preferences and get better at their preferred approaches. Without explicit tracking, a team's coverage narrows toward those techniques while total volume keeps rising — research on human red teaming has specifically highlighted the risk of repetitive testing on the same concepts. Categories that are easy to probe accumulate examples faster than the ones that require domain expertise, so a dataset can be large and still thin in exactly the areas that carry the most regulatory risk. And any dataset that becomes public, or is used repeatedly against models that later train on similar distributions, decays as a measurement instrument; programmes need a rotation policy and a held-out set that is never published.
The remedy for all three is the discipline used in any well-run annotation programme with defined recruitment and certification: measure coverage per category, per technique and per language rather than in aggregate, and treat a rising total with flat category coverage as a warning rather than progress.
Who does the work of building these datasets, and what do they need?
Building adversarial safety datasets means paying people to spend their working days generating and reading content designed to be harmful, and that carries psychological risk a standard annotation workflow does not account for.
This is routinely omitted from methodology guides. The regulatory environment is beginning to reflect it: Kenya's draft AI policy, published for consultation in 2026, targets data annotation, content moderation and AI quality evaluation roles specifically, and proposes mandatory psychosocial support alongside written contracts, pay transparency and grievance mechanisms. Academic work published at CHI 2026 documents the precarity of this workforce in some markets.
The operational implications: exposure limits and rotation rather than continuous assignment to the most distressing categories; access to support, provided rather than signposted; informed consent about content type at recruitment, not discovered on day one; and named ownership of wellbeing, so it is somebody's job rather than everybody's assumption. This connects to a straightforward operational argument alongside the ethical one — red teaming quality depends on domain expertise and accumulated familiarity with a model's behaviour, and fair consent and pay practices for data contributors are also what keeps experienced red teamers in the role long enough to build that familiarity. Sustained adversarial testing requires dedicated expertise rather than one-off engagements, because teams need time to learn system behaviour, develop novel attacks and track evolving threats. Burning out experienced red teamers destroys exactly the capability the programme depends on.
How does Lifewood approach red-teaming data?
Lifewood runs red teaming and safety evaluation as part of its AI data services, alongside RLHF and preference data collection, across 50+ languages with human-in-the-loop review as the operating discipline.
Two observations from that vantage, offered as observations rather than pitch. The first is that most organisations commissioning red teaming ask for volume and receive volume. The harder and more useful specification is coverage: how many categories, how many techniques, how many languages, with what per-cell depth, and what is deliberately excluded. A vendor who cannot produce a coverage matrix is selling prompt count — a distinction worth checking against any enterprise LLM training data partner's actual reporting, not its pitch deck.
The second concerns language, which is where footprint is relevant. Producing genuine adversarial data in Kiswahili or Sylheti requires red teamers who are native speakers with cultural fluency, because the attacks that work in a language are built on its idiom, its indirection and its cultural reference points. Translating English attack prompts produces material that tests translation, not the language — a recruitment and delivery problem before it is a methodology problem, and the reason this work runs through in-region teams rather than centrally, with results checked through the same data validation discipline applied to any other training dataset.
How should a team get started?
Start by writing the taxonomy before looking at any dataset, derived from your own deployment context and risk assessment, then compare it to established taxonomies and note deliberately what has been excluded and why.
Build a held-out set that is never published — the only instrument that survives models training on public benchmarks. Track technique diversity explicitly, not just prompt volume, and report coverage per category, per technique and per language, since aggregate counts conceal exactly the gaps that matter. Grade severity rather than simply flagging, because remediation priority requires it and so does defensible documentation. Test in every language of deployment, treating each as a separate programme rather than a translation exercise. Build the wellbeing framework before recruiting, not after the first person struggles. And document the methodology to the standard a regulator would read, because under the EU AI Act one may — a standard worth benchmarking against the practices covered in comparisons of large-scale training data providers.