AI Red-Teaming

What red-teaming actually means for AI

The term comes from military and cybersecurity practice: a 'red team' plays the adversary deliberately, attacking a system the way a real attacker would, so defenders can find and fix weaknesses before someone malicious does. AI red-teaming applies the same logic to a model or AI product: dedicated effort to find inputs, prompts, or usage patterns that cause it to produce harmful, unsafe, biased, or otherwise unintended outputs — before real users or bad actors find them in production.

This is distinct from standard evaluation. A benchmark asks 'how well does the model perform on a known, fixed set of tasks?' Red-teaming asks 'what can we make this model do that it shouldn't?' — an open-ended, adversarial search rather than a scored test on fixed questions.

What red-teamers actually probe for

Red-teaming efforts typically target several categories of failure: generating genuinely harmful content (instructions for violence, weapons, or other serious harms), producing biased or discriminatory outputs, leaking sensitive information the model was trained on or given in context, being manipulated via jailbreaks (prompts specifically engineered to bypass safety training through role-play framing, obfuscation, or multi-turn social engineering), falling for prompt injection in agentic contexts, and more mundane but still serious failures like confidently stating false information in high-stakes domains (medical, legal, financial) or taking unsafe actions when given tool access.

The scope expands with the system's capability: a plain chatbot's red-teaming surface is mostly about output content; an agentic system with tool access and the ability to take real-world actions has a much larger surface, because a successful attack can translate into an actual harmful action, not just a bad sentence.

Human red-teaming vs automated red-teaming

Human red-teaming uses people — sometimes domain experts (security researchers, medical professionals, policy specialists), sometimes crowdworkers specifically trained in adversarial prompting — to creatively probe a model. Humans are good at the kind of contextual, socially-aware creativity that produces genuinely novel attack framings, and at judging nuanced harms that require real-world judgment to evaluate. The downside is that human red-teaming is slow, expensive, and limited in coverage — a team of even dozens of people can only try so many attack variations before a launch deadline.

Automated red-teaming uses other AI models (sometimes the model being tested itself, sometimes a separate dedicated adversarial model) to generate large volumes of candidate attack prompts automatically, test them against the target model, and flag the ones that succeed. This scales far beyond what human effort alone could cover and can run continuously, but it tends to be less creative than skilled humans at genuinely novel attack strategies and can overfit to patterns the automated generator happens to favor.

Human red-teaming vs automated red-teaming

Why it became standard pre-launch practice

As LLMs moved from research demos to products used by millions of people — and increasingly into agentic systems with real-world tool access — the cost of a single serious safety failure found by the public, press, or a malicious actor after launch became far higher than the cost of finding it beforehand. Red-teaming is now typically embedded directly into the development lifecycle rather than bolted on at the end: run continuously during training and fine-tuning, intensified before major releases, and often supplemented by external red-teaming (independent researchers or specialized firms given early access specifically to try to break the system) precisely because an organization testing its own product has blind spots an outsider won't share.

In practice, most serious AI labs and AI product teams treat red-teaming results as a gating input to launch decisions, not just a nice-to-have audit — a system with unresolved high-severity red-team findings generally doesn't ship until those findings are mitigated or explicitly accepted as a known, bounded risk.

What's next

Red-teaming closes the loop on the safety and security themes running through this part of the track — from prompt injection to constitutional alignment to adversarial testing. Together, these techniques reflect a broader shift: as models become more capable and more autonomous, understanding what can go wrong, and systematically checking for it, has become as central to AI development as improving raw capability.