Short answer. Three decisions constrain a red-teaming dataset before any prompt is written: whose taxonomy you adopt, whether you build, borrow or collect, and how you split automated against human testing. The taxonomy matters most, because it determines what can be found at all — whatever sits outside it is invisible rather than absent, and CHI 2026 research found the construction approach itself shapes how practitioners conceive of risk. This is now a compliance question too: the EU AI Act reached full enforcement in August 2026, with Article 55 requiring documented adversarial testing for GPAI models carrying systemic risk.
Datasets for LLM Red Teaming?
There is a finding in a 2026 CHI paper on red teaming practice that I have not been able to stop thinking about, because it reframes the entire exercise.
Researchers interviewed practitioners about how they build adversarial datasets and found that the approach taken to dataset creation shaped their conceptualisation of risk. Teams building on existing datasets inherited the definitions of harm already embedded in the source material. Teams building from scratch selected their own categories.
Read that again, because the implication is uncomfortable. The taxonomy is not a neutral container for findings. It determines what can be found. A red teaming programme with fourteen harm categories will discover harms in fourteen categories. Whatever falls outside them is not a negative result, it is invisible.
That is the honest starting point for this topic, and it explains why so much red teaming produces reassuring reports about models that later fail in production.
A note on scope before I continue. This piece is about how safety datasets are built, staffed and governed as a data operation. It is not a catalogue of attack techniques, and I have deliberately kept it at the level of methodology rather than method. The organisations doing this work well publish their processes, not their payloads.
Why this is now a compliance requirement, not a research exercise
The regulatory position hardened considerably.
The EU AI Act reached full enforcement in August 2026. Article 55 requires documented adversarial testing for general-purpose AI models with systemic risk. Article 9 mandates risk management systems including testing procedures throughout the lifecycle. Non-compliance carries penalties up to €35 million or 7% of global annual turnover.
In the United States, the NIST AI Risk Management Framework MEASURE function formalises adversarial testing as standard practice, and federal procurement guidance increasingly references red teaming as a requirement.
So the question for most organisations has moved from whether to do this to how to do it in a way that produces defensible documentation. And "documented adversarial testing" means the dataset, the taxonomy, the coverage, the methodology and the results, not a summary paragraph saying testing occurred.
The three decisions that determine everything downstream
The CHI research identifies three critical moments in this work: defining and framing the task, developing the adversarial dataset, and evaluating models against it. Each carries a decision that constrains everything after it.
Decision one: whose taxonomy?
Reusing an established taxonomy gives methodological consistency and comparability with published results. It also means inheriting someone else's definition of harm, built for someone else's deployment context.
The alternative is building categories from your own risk assessment, which produces more relevant coverage and makes your results incomparable to anyone else's.
Neither is wrong. What is wrong is making the choice by default, which is what happens when a team downloads a public dataset and starts running it without examining what it does and does not classify as harmful.
A concrete illustration of how taxonomies embed assumptions: analysis of one widely used safety dataset noted that responses demonstrating human-like emotion or behaviour were labelled as safe, which leaves emotional manipulation risks entirely outside the measurement. That is not an error in the annotation. It is a category that was never drawn.
Decision two: build, borrow or collect?
Practitioners described three sources, with a clear hierarchy of perceived value. Reusing well-known datasets was seen as a way to ensure methodological consistency. Building from scratch, or collecting real human-LLM interactions, was perceived as producing unique and therefore more valuable data.
There is a hard reason that perception is correct, and it is the most important operational fact in this field: public adversarial datasets have become less reliable because models are increasingly trained to pass them. A benchmark that enters the training distribution stops measuring safety and starts measuring memorisation.
Which means a public dataset tells you your model handles known attacks. It tells you nothing about novel ones, and novel ones are the entire point.
Decision three: automated, human, or both?
Automated red teaming frameworks reportedly discover vulnerabilities at 3.9 times the rate of manual expert testing while covering multiple threat categories simultaneously. That is a real efficiency argument and it should not be dismissed.
But the framing that holds up across sources is that automated tools provide breadth and human red teamers provide depth, novel attack discovery and domain expertise. Anthropic's Policy Vulnerability Testing and OpenAI's external red teaming network are structured differently but rest on the same premise: domain-expert humans construct probes no automated attacker would think to write.
What the frontier labs actually publish about their process
This is worth knowing because it sets a reference standard, and because the numbers reveal what serious coverage looks like.
The GPT-4o System Card reports over 100 external red teamers across 29 countries speaking 45 different languages. The Operator System Card, covering OpenAI's web-browsing agent, reports 20 countries and 24 languages for that surface specifically. OpenAI's methodology is published in "Approach to External Red Teaming for AI Models and Systems" by Ahmad and colleagues.
Anthropic's Policy Vulnerability Testing describes in-depth qualitative testing with external subject-matter experts on specific Usage Policy topics.
Two things stand out. First, the scale of external involvement: this is not an internal team running a checklist. Second, and more striking for anyone building a programme, language and country coverage is reported as a headline metric.
Forty-five languages is a deliberate statement that safety behaviour varies by language and that testing in one language does not generalise.
Most enterprise red teaming programmes I have seen documented test in English only, and then report a safety result as though it applied to the deployment.
The multilingual gap, which is the largest hole in most programmes
This deserves its own section because the evidence is unusually clear and the implications are underappreciated.
Safety alignment is trained, not inherited. A guardrail exists in the languages it was taught in, which means a model can be strict in English and permissive elsewhere.
Researchers at Brown University demonstrated this by taking unsafe English prompts, translating them into low-resource languages such as Zulu using freely available translation, and sending them to GPT-4. The approach produced harmful responses roughly 80% of the time, against a model that refused the same requests in English.
The picture has evolved rather than resolved. A 2026 study testing African languages including Kiswahili, isiXhosa and isiZulu found that straightforward translation attacks no longer succeed as easily, which is genuine progress, but that conversations spread across multiple turns still succeed at high rates. Separate evaluation across 79 languages found unsafe response rates rising by as much as 25 percentage points as prompts moved from English into low-resource languages.
The strategic point that gets missed: this is not only a problem for speakers of those languages. A guardrail that fails in any language is a guardrail anyone can route around using a free translation tool. Multilingual safety data is a security control, not a diversity initiative.
The field is responding with language-specific safety datasets, including work on Albanian and a growing body of similar efforts, but coverage remains thin relative to the number of languages models are deployed in.
What a usable dataset actually contains
A collection of adversarial prompts is not a dataset. The 2022 Anthropic red team release illustrates the gap: 38,961 red team attacks spanning twenty categories, which is substantial scale, but analysis noted that the absence of labelled responses reduced its effective utilisation for both automated red teaming and evaluation.
A dataset that supports both evaluation and training needs, per record:
The prompt or conversation, including the full multi-turn sequence where relevant, since multi-turn is where much of the current risk sits.
The model response, verbatim.
A harm category label from a documented taxonomy, with the taxonomy version recorded.
A severity grading, because "unsafe" is not one thing and remediation priority depends on degree.
A success or failure judgement on whether the attempt achieved its objective.
Attack technique classification, tracked so that technique diversity can be measured rather than assumed. One practitioner organisation notes that it trains red teamers across categories of adversarial technique and systematically tracks technique diversity to ensure comprehensive coverage across engagements, which is a discipline worth copying: without it, a team drifts toward the techniques individuals find most productive and coverage silently narrows.
Language and locale.
Annotator identity and adjudication history, for the same reasons any annotation programme needs them.
On labelling philosophy, the guidance from practitioners is consistent: conservative annotation, where safety labels err toward caution, produces the most reliable training data for defensive systems. A false positive costs a refused benign request. A false negative costs the thing the programme exists to prevent.
Coverage, and how programmes quietly narrow
Three failure patterns recur, all of which look like success in a dashboard.
Technique concentration. Individual red teamers develop preferences and get better at their preferred approaches.
Without explicit tracking, a team's coverage narrows toward those techniques while total volume keeps rising. Research on human red teaming has specifically highlighted the risk of repetitive testing on the same concepts.
Category imbalance. The categories that are easy to probe accumulate examples faster than the ones that require domain expertise. A dataset can be large and still thin in exactly the areas that carry the most regulatory risk.
Benchmark contamination over time. Any dataset that becomes public, or that is used repeatedly against models that later train on similar distributions, decays as a measurement instrument. Programmes need a rotation policy and a held-out set that is never published.
The remedy for all three is the same discipline used in any annotation programme: measure coverage per category, per technique and per language rather than in aggregate, and treat a rising total with flat category coverage as a warning rather than progress.
The people doing this work
This section matters and is routinely omitted from methodology guides.
Building adversarial safety datasets means paying people to spend their working days generating and reading content designed to be harmful. That is a materially different job from image annotation, and it carries psychological risk that a standard annotation workflow does not account for.
The regulatory environment is beginning to reflect this. Kenya's draft AI policy, published for consultation in 2026, targets data annotation, content moderation and AI quality evaluation roles specifically, and proposes mandatory psychosocial support alongside written contracts, pay transparency and grievance mechanisms.
Academic work published at CHI 2026 documents the precarity of this workforce in some markets.
The operational implications for anyone running or commissioning this work:
Exposure limits and rotation rather than continuous assignment to the most distressing categories.
Access to support, provided rather than signposted.
Informed consent about content type at recruitment, not discovered on day one.
Named ownership of wellbeing, so it is somebody's job rather than everybody's assumption.
There is a straightforward operational argument alongside the ethical one. Red teaming quality depends on domain expertise and accumulated familiarity with a model's behaviour. Sustained adversarial testing requires dedicated expertise rather than one-off engagements, because teams need time to learn system behaviour, develop novel attacks and track evolving threats. Burning out experienced red teamers destroys exactly the capability the programme depends on.
Where we sit in this
Declaring the interest plainly: Lifewood runs red teaming and safety evaluation as part of its AI data services, alongside RLHF and preference data work, across 50-plus languages with human-in-the-loop review as the operating discipline.
Two observations from that vantage, offered as observations rather than pitch.
The first is that most organisations commissioning red teaming ask for volume and receive volume. The harder and more useful specification is coverage: how many categories, how many techniques, how many languages, with what per-cell depth, and what is deliberately excluded. A vendor who cannot produce a coverage matrix is selling prompt count.
The second concerns language, which is where our own footprint is relevant. Producing genuine adversarial data in Kiswahili or Sylheti requires red teamers who are native speakers with cultural fluency, because the attacks that work in a language are built on its idiom, its indirection and its cultural reference points. Translating English attack prompts produces material that tests translation, not the language. That is a recruitment and delivery problem before it is a methodology problem, and it is why we run this work through in-region teams rather than centrally.
Where to start
Write the taxonomy before looking at any dataset, derived from your deployment context and risk assessment. Then compare it to established taxonomies and note deliberately what you have excluded and why.
Build a held-out set you never publish. This is the only instrument that survives models training on public benchmarks.
Track technique diversity explicitly, not just prompt volume.
Report coverage per category, per technique and per language. Aggregate counts conceal exactly the gaps that matter.
Grade severity, do not just flag. Remediation priority requires it and so does defensible documentation.
Test in every language you deploy in, treating each as a separate programme rather than a translation exercise.
Build the wellbeing framework before recruiting, not after the first person struggles.
Document the methodology to the standard a regulator would read, because under the EU AI Act one may.
Key takeaways
- CHI 2026 research found that the approach taken to dataset creation shapes practitioners' conceptualisation of risk.
- The taxonomy determines what can be found; whatever falls outside it is invisible rather than absent.
- The EU AI Act reached full enforcement in August 2026. Article 55 requires documented adversarial testing for GPAI models with systemic risk, Article 9 mandates lifecycle testing procedures, and penalties reach €35 million or 7% of global turnover.
- The NIST AI RMF MEASURE function formalises adversarial testing in the United States.
- Three decisions constrain everything downstream: whose taxonomy, whether to build, borrow or collect, and the balance of automated to human testing.
- Reusing established taxonomies inherits someone else's definition of harm. One widely used dataset labelled human-like emotional responses as safe, leaving emotional manipulation outside measurement entirely.
- Public adversarial datasets have become less reliable because models are increasingly trained to pass them. A benchmark inside the training distribution measures memorisation, not safety.
- Automated frameworks reportedly find vulnerabilities at 3.9 times the rate of manual testing, but automation provides breadth while human red teamers provide depth and novel attack discovery.
- The GPT-4o System Card reports over 100 external red teamers across 29 countries and 45 languages. Anthropic runs Policy Vulnerability Testing with external subject-matter experts.
- Language coverage is reported as a headline metric by frontier labs because safety behaviour does not generalise across languages.
- Brown University research found translating unsafe prompts into low-resource languages produced harmful responses roughly 80% of the time. A 2026 African languages study found translation attacks now largely fail but multi-turn conversations still succeed at high rates. Evaluation across 79 languages found unsafe response rates rising up to 25 percentage points.
- A guardrail that fails in any language is a vulnerability for every user, because translation is free.
- Anthropic's 2022 release of 38,961 red team attacks across 20 categories had limited utility because responses were unlabelled. A usable record needs prompt, response, harm category, severity, success judgement, technique classification, language and annotator history.
- Conservative annotation erring toward caution produces the most reliable training data for defensive systems.
- Programmes narrow silently through technique concentration, category imbalance and benchmark contamination.
- Measure coverage per category, technique and language rather than in aggregate.
- Kenya's 2026 draft AI policy proposes mandatory psychosocial support for AI quality evaluation roles. Exposure limits, rotation, provided support and named wellbeing ownership are operational requirements, and burnout destroys the accumulated expertise the programme depends on.
Sources and further reading
- "Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation", Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, on the three critical moments and how dataset creation approach shapes risk conceptualisation
- LXT, "LLM Red Teaming: A 5-Phase Framework for AI Security", on EU AI Act Articles 55 and 9, penalty levels, the NIST AI RMF MEASURE function, the 3.9x automated discovery rate and conservative annotation guidance
- Kili Technology, "LLM Red Teaming in 2026: How Frontier Labs Test AI", on GPT-4o and Operator System Card red teamer and language counts, OpenAI's published methodology by Ahmad and colleagues, Anthropic's Policy Vulnerability Testing, and public dataset decay
- "Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs", arXiv, on the Ganguli et al. 38,961-attack dataset and the limitation of unlabelled responses, and on taxonomy gaps around human impacts
- "AART: AI-Assisted Red-Teaming with Diverse Data Generation", arXiv, on tester diversity and the risk of repetitive testing on the same concepts
- Innodata, "Model Safety, Evaluation and Red Teaming Solutions", on structured taxonomies of adversarial technique and systematic tracking of technique diversity across engagements
- "AlbanianLLMSafety: A Safety Evaluation Dataset for Large Language Models in Albanian", arXiv, on Llama Guard, customisable harm taxonomies and language-specific safety dataset development
- Lifewood, AI data services including red teaming, RLHF and human-in-the-loop quality assurance
- Note on scope: this article addresses red teaming as a data operations and governance discipline. It deliberately does not describe attack techniques or include adversarial prompts. Multilingual safety findings referenced here are drawn from the Brown University low-resource jailbreak research and subsequent 2026 evaluations discussed in earlier articles in this series.