# OpenAI's Approach to External Red Teaming for AI Models and Systems

arXiv: https://arxiv.org/abs/2503.16431 · HTML: https://mlsanity.com/paper/2503.16431/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 21 of 21)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models](https://mlsanity.com/paper/2406.18510/) (2024)
- [Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets](https://mlsanity.com/paper/2106.10328/) (2021)
- [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020)
- [Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models](https://mlsanity.com/paper/2102.02503/) (2021)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [Rule Based Rewards for Language Model Safety](https://mlsanity.com/paper/2411.01111/) (2024)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support](https://mlsanity.com/paper/2112.05675/) (2021)
- [Towards interactive evaluations for interaction harms in human-AI systems](https://mlsanity.com/paper/2405.10632/) (2024)
- [Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities](https://mlsanity.com/paper/2207.11345/) (2022)
- [A Hazard Analysis Framework for Code Synthesis Large Language Models](https://mlsanity.com/paper/2207.14157/) (2022)
- [Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming](https://mlsanity.com/paper/2311.06237/) (2023)
- [Confidence-Building Measures for Artificial Intelligence: Workshop Proceedings](https://mlsanity.com/paper/2308.00862/) (2023)

## Built on by (most cited first, 28 of 28)

- [The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections](https://mlsanity.com/paper/2510.09023/) (2025)
- [BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions](https://mlsanity.com/paper/2510.05318/) (2025)
- [Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmas](https://mlsanity.com/paper/2505.14633/) (2025)
- [PurpCode: Reasoning for Safer Code Generation](https://mlsanity.com/paper/2507.19060/) (2025)
- [Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming](https://mlsanity.com/paper/2601.18076/) (2026)
- [Distributional AGI Safety](https://mlsanity.com/paper/2512.16856/) (2025)
- [Watermarking Without Standards Is Not AI Governance](https://mlsanity.com/paper/2505.23814/) (2025)
- [In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI](https://mlsanity.com/paper/2503.16861/) (2025)
- [Red Teaming Contemporary AI Models: Insights from Spanish and Basque Perspectives](https://mlsanity.com/paper/2503.10192/) (2025)
- [Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Experts](https://mlsanity.com/paper/2510.07239/) (2025)
- [Red Teaming AI Red Teaming](https://mlsanity.com/paper/2507.05538/) (2025)
- [From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teaming](https://mlsanity.com/paper/2509.11398/) (2025)
- [Review of Hallucination Understanding in Large Language and Vision Models](https://mlsanity.com/paper/2510.00034/) (2025)
- [Longitudinal Monitoring of LLM Content Moderation of Social Issues](https://mlsanity.com/paper/2510.01255/) (2025)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [Beyond Abstract Compliance: Operationalising trust in AI as a moral relationship](https://mlsanity.com/paper/2601.22769/) (2026)
- [Buy versus Build an LLM: A Decision Framework for Governments](https://mlsanity.com/paper/2602.13033/) (2026)
- [Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center](https://mlsanity.com/paper/2506.22523/) (2025)
- [PolicyPad: Collaborative Prototyping of LLM Policies](https://mlsanity.com/paper/2509.19680/) (2025)
- [Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimization](https://mlsanity.com/paper/2601.13359/) (2026)
- [Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation](https://mlsanity.com/paper/2602.18483/) (2026)
- [Red-Teaming Claude Opus and ChatGPT-based Security Advisors for Trusted Execution Environments](https://mlsanity.com/paper/2602.19450/) (2026)
- [ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward System](https://mlsanity.com/paper/2604.18789/) (2026)
- [AVISE: Framework for Evaluating the Security of AI Systems](https://mlsanity.com/paper/2604.20833/) (2026)
- [FlashRT: Towards Computationally and Memory Efficient Red-Teaming for Prompt Injection and Knowledge Corruption](https://mlsanity.com/paper/2604.28157/) (2026)
- [Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolution](https://mlsanity.com/paper/2605.13411/) (2026)
- [Position: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safety](https://mlsanity.com/paper/2607.05407/) (2026)
- [A Threshold Exceedance Framework for CBRN Uplift Evaluation in Frontier Language Models](https://mlsanity.com/paper/2607.12200/) (2026)

## Abstract (arXiv)

Red teaming has emerged as a critical practice in assessing the possible risks of AI models and systems. It aids in the discovery of novel risks, stress testing possible gaps in existing mitigations, enriching existing quantitative safety metrics, facilitating the creation of new safety measurements, and enhancing public trust and the legitimacy of AI risk assessments. This white paper describes OpenAI's work to date in external red teaming and draws some more general conclusions from this work. We describe the design considerations underpinning external red teaming, which include: selecting composition of red team, deciding on access levels, and providing guidance required to conduct red teaming. Additionally, we show outcomes red teaming can enable such as input into risk assessment and automated evaluations. We also describe the limitations of external red teaming, and how it can fit into a broader range of AI model and system evaluations. Through these contributions, we hope that AI developers and deployers, evaluation creators, and policymakers will be able to better design red teaming campaigns and get a deeper look into how external red teaming can fit into model deployment and evaluation processes. These methods are evolving and the value of different methods continues to shift as the ecosystem around red teaming matures and models themselves improve as tools for red teaming.