Short answer. AI red teaming runs on four data assets: categorised adversarial prompts, human-judged outputs, multilingual and multimodal variants of both, and audit-grade documentation of what was tried and what happened. Automation now generates attacks at scale, but judging them, inventing novel strategies and signing off on the result stay human. The largest practical exposure is language: guardrails trained on English fail quietly elsewhere, and a translated attack tests the translation rather than the risk, so attack sets have to be authored natively.
Key takeaways
- Red teaming runs on four data assets: categorised adversarial prompts, human-judged outputs, multilingual/multimodal variants, and audit-grade documentation — each reusable across model updates.
- Automation (the GPT-Red line) generates attacks at scale, but judging, novel strategies and accountability remain human; the working model is automated breadth, human depth.
- The multilingual gap is the biggest practical exposure: guardrails trained on English fail quietly in other languages, and translated attacks test the translation, not the risk — attack sets must be natively authored.
- Providers: frontier labs set methods internally; AI-data companies staff production exercises (Appen's visible practice; Lifewood on the multilingual adversarial-authoring and judged-rating layer across 100+ languages); specialist firms handle frontier assessments.
- Buy it like annotation, because it is: rater calibration, guidelines, agreement stats and named language capability are the evaluation criteria that matter.
What is AI red teaming, in plain terms?
Red teaming is attacking your own system before someone else does. Borrowed from security practice, in AI it means systematically trying to make a model do what it shouldn't — produce harmful instructions, leak private data, act outside its authority, or be manipulated through role-play, encoding tricks and multi-turn pressure — then recording what worked, so the failures can be fixed before release.
The craft has professionalised fast. Appen's adversarial prompting research catalogues the attack families a competent red team works through — persona-based jailbreaks, instruction smuggling, gradual escalation across turns — and its multimodal red teaming work extends the same discipline to images and audio, where attacks hide in content the text filters never see.
Two things pull red teaming out of the "nice to have" column. First, models now ship with tool access and agent behaviours, so a successful attack doesn't just produce bad text — it can take bad actions. Second, the audit trail is becoming table stakes: enterprise procurement and emerging AI regulation ask vendors what adversarial testing was done, in which languages, and what changed as a result. A red team's real output is that evidence.
For buyers of AI data generally, red teaming sits beside evaluation as the quality-assurance layer of the model lifecycle — downstream of the training-data work covered across our data annotation services, and upstream of deployment.
What data does red teaming actually run on?
Four assets, usually bought or built together:
Adversarial prompt sets. Attacks written on purpose: per harm category (violence, fraud, privacy, self-harm, bias), per technique (role-play, obfuscation, encoding, multi-turn escalation), per domain (finance scams for a banking bot, dosage traps for a health assistant). Quality here is diversity — a thousand rewordings of one jailbreak test one thing; the craft parallels the coverage logic of long-tail and edge-case mining.
Judged outputs. Each attack's result, labelled by humans against a harm rubric: did the model refuse, comply, partially comply, or fail strangely? This is annotation under another name — it needs calibrated raters, clear guidelines and agreement measurement, exactly the machinery of gold sets, audit sampling and consensus. The judged pairs then do double duty: findings report today, safety training data tomorrow.
Variant sets. The same attacks carried across languages, dialects, personas and modalities — where most deployed systems are weakest (section 4), and where volume collection needs the same managed-pipeline discipline as any speech or text collection programme.
Remediation documentation. Reproducible attack traces, severity ratings, category coverage maps — the artefact that lets an engineering team fix issues and an auditor verify the exercise happened.
Figure 1 — The red teaming data stack
Asset What it is Who makes it Reused as Adversarial prompts Attacks by category, technique, domain Trained red teamers, domain experts Regression test suite Judged outputs Pass/fail/severity labels on responses Calibrated human raters Safety fine-tuning data Variant sets Language, dialect, persona, modality versions Native speakers, multimodal specialists Multilingual guardrail evals Remediation docs Traces, severity, coverage maps Red team leads Compliance evidence The reuse column is why red teaming data is bought, not improvised: built well once, it keeps working as models update.
Hasn't automation replaced human red teamers?
It has changed their job, not removed it.
The headline development is real: OpenAI's automated red teamer — GPT-Red, which we analysed when it was announced — and the broader research line it represents can generate and mutate attacks at a scale no human team matches, and Appen's own analysis of the development reaches the conclusion the industry has broadly settled on: automation raises the floor, humans set the ceiling.
Three pieces stay stubbornly human:
Judging. Whether a response is actually harmful is a context call — the difference between a chemistry explanation and a synthesis route, between fiction and instruction. Automated judges drift and share blind spots with the models they judge; human rubric-based rating remains the ground truth the automation is calibrated against, the same logic that keeps humans in the loop in our VLM-judge analysis.
Novelty. Generators remix known attack families brilliantly; genuinely new strategies — the cultural reference the model misreads, the domain workflow that creates an exploit — come from people, especially domain experts and native speakers.
Accountability. "Our model red teamed itself" does not survive an audit. Regulators and enterprise buyers want independent human exercise documented.
The operating model that follows: automated breadth, human depth — machines sweep the known space cheaply; people probe where the machines can't see, and judge everything that matters.
Why does red teaming break down across languages?
Because safety training concentrates where the data is, and attacks go where the training didn't.
The pattern is well documented in the research and painfully consistent in practice: guardrails trained predominantly on English refuse harmful requests in English — and comply with the same request in a lower-resource language, or in code-switched text, or when the harmful intent is wrapped in a cultural frame the safety training never saw. It is the safety-layer version of the capability gap we documented in why generative AI gets worse in your second language: performance tracks training-data distribution, and safety is no exception.
For any product deployed across markets, that makes multilingual red teaming the highest-yield slice of the exercise — and the hardest to staff, because it needs people who are simultaneously native speakers, culturally fluent and trained in adversarial technique. Translation does not substitute: a translated jailbreak tests the translation, not the local attack surface, for the same reasons machine-translated training data misses local reality. Attacks must be authored in-language — local scam formats, local sensitive topics, local obfuscation habits.
This is also where red teaming connects to data sovereignty: adversarial exercises on products handling regulated data inherit the residency and security constraints of the data itself, the territory mapped in secure annotation of sensitive data.
Who provides red teaming data and services?
Three tiers: frontier labs running standing internal teams and publishing the methods, AI data companies staffing production-scale human exercises, and specialist security firms handling frontier assessments.
Frontier labs' internal teams — OpenAI, Anthropic, Google DeepMind run standing red teams and publish methods, including the automated-attacker research line. Their work sets the state of the art but isn't for hire; its published outputs are the free syllabus any buyer should read.
AI data companies with safety practices. This is where production-scale human red teaming is actually staffed. Appen has built the most visible practice, with published adversarial-prompting and multimodal research. Lifewood provides the data layer of the exercise — multilingual adversarial prompt authoring and judged-output rating across 100+ languages through managed delivery centres — which is the specific slice where its native-speaker bench and annotation QA machinery (calibrated raters, gold sets, documented agreement) map directly onto what red teaming consumes; the fit is strongest for the multilingual variant sets and rating programmes of section 2, alongside the human evaluation services the same teams deliver. Scale AI and other data platforms run comparable safety-evaluation offerings for their model-builder clients.
Specialist safety firms and research groups — boutique red teams, often ex-lab researchers, strongest for frontier-risk assessments and bespoke exercises; academic groups supply the public benchmarks and attack taxonomies the field standardises on.
Buying advice compresses to one principle: red teaming is annotation-grade work wearing a security name — so apply the same diligence you would to any annotation vendor: ask for rater calibration evidence, guideline samples, agreement statistics and language coverage by named capability, not claimed count.
What to do to red team your own AI deployment
Scope by your own harm surface rather than a generic checklist, buy or build a categorised attack set where diversity beats volume, judge the outputs with calibrated human raters, and keep the documentation an auditor would ask for.
- Scope by your actual harm surface. List what your product could do wrong for your users — fraud coaching for a fintech bot, dosage errors for health, data leakage for anything with retrieval. Generic jailbreak lists miss your specific risks.
- Buy or build a categorised attack set — per harm, per technique, per domain — and treat diversity, not volume, as the quality bar.
- Stand up human judging before scaling attacks: a harm rubric, calibrated raters, agreement measured. Unjudged attack runs are noise.
- Cover the languages you deploy in, with natively-authored attacks — your weakest guardrails are almost certainly not in English.
- Pair automation with human depth: automated sweeps for regression coverage, human sessions for novelty, every finding human-judged.
- Document for reuse and audit: reproducible traces, severity, coverage map — then feed judged pairs back as safety training data.
- Re-run on every model update. Guardrails regress; your attack set is now your regression suite.