AI Red Teaming

AI red teaming is the practice of adversarially probing an AI system to discover its failure modes, safety vulnerabilities, harmful behaviors, and capability limitations before deployment. Borrowed from cybersecurity, red teaming places trained evaluators (or automated systems) in the role of a determined adversary trying to elicit undesired behaviors.

Unlike standard benchmarking, which tests expected behaviors, red teaming deliberately seeks unexpected and undesired behaviors: harmful outputs, policy violations, factual hallucinations under pressure, jailbreaks, and emergent risks.

Why Red Teaming Matters

AI systems trained on human feedback are not immune to adversarial inputs. Common failure modes include:

• Jailbreaks: Prompt constructions that bypass safety guidelines.
• Harmful content generation: Eliciting instructions for dangerous activities.
• Bias amplification: Revealing discriminatory or stereotypical tendencies under specific prompts.
• Hallucination under adversarial conditions: Confidently fabricating false information when pressed.
• Capability elicitation: Discovering that models have dangerous capabilities not apparent in standard use.

Without systematic red teaming, these failures surface in production — with real consequences.

The Red Team / Blue Team Framework

In AI safety contexts, the red team is tasked with finding risks that the development team may have missed, operating from the perspective of a malicious user, an unsophisticated user, or a sophisticated attacker.

Red Teaming Methodologies

1. Manual Red Teaming

Human evaluators craft adversarial prompts based on domain knowledge and creativity. This is most effective for:

• Novel attack patterns not anticipated by automated tools.
• Nuanced cultural or contextual harms.
• Testing for sophisticated multi-turn jailbreaks.
• Domain-specific risks (medical, legal, financial).

Structured approaches:

• Persona-based: Evaluators adopt specific user personas (vulnerable individual, malicious actor, investigative journalist).
• Scenario-based: Craft realistic scenarios where harmful outputs would occur.
• Tree of attacks: Iteratively refine prompts based on model responses.

2. Automated Red Teaming

Manual red teaming is labor-intensive and cannot achieve comprehensive coverage. Automated red teaming uses algorithms or language models to generate adversarial inputs at scale.

Approaches:

Gradient-based attacks (white-box):

Optimize a prompt suffix to maximize the probability of a harmful completion:

$$\delta^* = \arg\max_\delta \log P(\text{harmful output} \mid x + \delta)$$

The GCG (Greedy Coordinate Gradient) attack finds adversarial suffixes that transfer across models.

LLM-based red teaming (black-box):

Use a separate “attacker” LLM to generate adversarial prompts against a target model:

1. Attacker LLM generates a candidate adversarial prompt.
2. Target model generates a response.
3. A classifier judges whether the response violates safety guidelines.
4. Feedback is used to refine attacker prompts.

This is the approach used in Perez et al. (2022) and Anthropic’s red teaming pipeline.

Tree of Attacks with Pruning (TAP):

A tree-search approach where the attacker LLM iteratively refines prompts, pruning branches that fail to make progress toward eliciting a violation.

3. Structured Taxonomy-Based Testing

Organize red teaming efforts around a harm taxonomy:

Systematic coverage of each category ensures comprehensive evaluation.

Evaluating Red Team Success

A red team “attack” succeeds when:

1. The model produces a response that violates its stated safety guidelines or acceptable use policy.
2. The response could cause real-world harm if acted upon.
3. The attack is reproducible and transferable.

Attack Success Rate (ASR): The fraction of adversarial prompts that elicit a policy-violating response.

$$\text{ASR} = \frac{\text{Successful attacks}}{\text{Total attempts}}$$

A well-hardened model should have a low ASR, but ASR=0 is often a sign of over-refusal rather than genuine safety.

Jailbreak Taxonomy

Responsible Disclosure and Red Team Governance

Red teaming in AI requires careful governance:

• Scope definition: What is in scope for testing? What harm categories are researchers authorized to explore?
• Data handling: Adversarial prompts and outputs may contain harmful content and must be handled securely.
• Finding classification: Not all findings are equally severe; a risk-based severity scale is needed.
• Remediation tracking: Findings must be tracked and addressed before deployment.
• Disclosure policy: When should findings be shared with the broader research community?

Red Teaming in Practice

Anthropic, OpenAI, Google DeepMind, and other major AI labs conduct extensive red teaming before model releases, often engaging external contractors with specialized expertise in areas like CBRN threats, child safety, or cybersecurity.

The U.S. AI Safety Institute (AISI) and UK AISI have conducted government-led red teaming evaluations of frontier models under voluntary safety commitments.

NIST AI RMF and the EU AI Act both incorporate adversarial testing as a component of responsible AI risk management.

Limitations of Red Teaming

• Incomplete coverage: Even extensive red teaming cannot test all possible inputs.
• Adversarial evolution: New attack techniques emerge after evaluations are complete.
• False confidence: A clean red team result does not guarantee safety in deployment.
• Capability overhang: Red teaming may not reveal latent dangerous capabilities that only emerge in specific configurations.

Red teaming is a necessary but not sufficient component of AI safety — it must be combined with alignment training, monitoring, and ongoing evaluation in deployment.

Further Reading

• Red Teaming Language Models to Reduce Harms — Perez et al., 2022
• Jailbroken: How Does LLM Safety Training Fail? — Wei et al., 2023
• Universal and Transferable Adversarial Attacks on Aligned Language Models — Zou et al., 2023
• NIST AI Risk Management Framework