# Red Teaming AI Red Teaming

arXiv: https://arxiv.org/abs/2507.05538 · HTML: https://mlsanity.com/paper/2507.05538/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 10 of its 44 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 10 of 10)

- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [OpenAI's Approach to External Red Teaming for AI Models and Systems](https://mlsanity.com/paper/2503.16431/) (2025)
- [Lessons From Red Teaming 100 Generative AI Products](https://mlsanity.com/paper/2501.07238/) (2025)
- [Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming](https://mlsanity.com/paper/2311.06237/) (2023)
- [Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects](https://mlsanity.com/paper/2505.18893/) (2025)
- [Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI](https://mlsanity.com/paper/2409.15398/) (2024)
- [In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI](https://mlsanity.com/paper/2503.16861/) (2025)
- [Red Teaming Contemporary AI Models: Insights from Spanish and Basque Perspectives](https://mlsanity.com/paper/2503.10192/) (2025)
- [The Promise and Peril of Artificial Intelligence -- Violet Teaming Offers a Balanced Path Forward](https://mlsanity.com/paper/2308.14253/) (2023)
- [I'm Sorry Dave: How the old world of personnel security can inform the new world of AI insider risk](https://mlsanity.com/paper/2504.00012/) (2025)

## Built on by (most cited first, 3 of 3)

- [Distributional AGI Safety](https://mlsanity.com/paper/2512.16856/) (2025)
- [CIRCLE: A Framework for Evaluating AI from a Real-World Lens](https://mlsanity.com/paper/2602.24055/) (2026)
- [LLJ Cards: Best practices for the Use of LLMs as Judges](https://mlsanity.com/paper/2609.24516/) (2026)

## Abstract (arXiv)

Red teaming has evolved from its origins in military applications to become a widely adopted methodology in cybersecurity and AI. In this paper, we take a critical look at the practice of AI red teaming. We argue that despite its current popularity in AI governance, there exists a significant gap between red teaming's original intent as a critical thinking exercise and its narrow focus on discovering model-level flaws in the context of generative AI. Current AI red teaming efforts focus predominantly on individual model vulnerabilities while overlooking the broader sociotechnical systems and emergent behaviors that arise from complex interactions between models, users, and environments. To address this deficiency, we propose a comprehensive framework operationalizing red teaming in AI systems at two levels: macro-level system red teaming spanning the entire AI development lifecycle, and micro-level model red teaming. Drawing on cybersecurity experience and systems theory, we further propose a set of six recommendations. In these, we emphasize that effective AI red teaming requires multifunctional teams that examine emergent risks, systemic vulnerabilities, and the interplay between technical and social factors.