mlsanityData to 14 Oct 2026

OpenAI's Approach to External Red Teaming for AI Models and Systems

arXiv 2503.16431 ↗First posted 24 Jan 2025cs.CY cs.AI cs.CRcited by 28

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Red teaming has emerged as a critical practice in assessing the possible risks of AI models and systems. It aids in the discovery of novel risks, stress testing possible gaps in existing mitigations, enriching existing quantitative safety metrics, facilitating the creation of new safety measurements, and enhancing public trust and the legitimacy of AI risk assessments. This white paper describes OpenAI's work to date in external red teaming and draws some more general conclusions from this work. We describe the design considerations underpinning external red teaming, which include: selecting composition of red team, deciding on access levels, and providing guidance required to conduct red teaming. Additionally, we show outcomes red teaming can enable such as input into risk assessment and automated evaluations. We also describe the limitations of external red teaming, and how it can fit into a broader range of AI model and system evaluations. Through these contributions, we hope that AI developers and deployers, evaluation creators, and policymakers will be able to better design red teaming campaigns and get a deeper look into how external red teaming can fit into model deployment and evaluation processes. These methods are evolving and the value of different methods continues to shift as the ecosystem around red teaming matures and models themselves improve as tools for red teaming.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 21 of its 42 references that are in the map. First posted on arXiv 24 Jan 2025.

Builds on

21 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  2. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?29 Jan
  3. 2023Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming10 Nov
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  5. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 16

  1. 2024GPT-4o System Cardcited by 4,588
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  4. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  5. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  6. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  7. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 165
  8. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 160
  9. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
  10. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 138
  11. 2024Rule Based Rewards for Language Model Safetycited by 108
  12. 2021Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Supportcited by 63
4 more
  1. 2024Towards interactive evaluations for interaction harms in human-AI systemscited by 29
  2. 2022Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparitiescited by 28
  3. 2022A Hazard Analysis Framework for Code Synthesis Large Language Modelscited by 22
  4. 2023Confidence-Building Measures for Artificial Intelligence: Workshop Proceedingscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 13.

28 papers.

Sort:

By area:

  1. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  2. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 19
  3. 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 18
  4. 2025PurpCode: Reasoning for Safer Code Generationcited by 18
  5. 2026Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teamingcited by 15
  6. 2025Distributional AGI Safetycited by 12
  7. 2025Watermarking Without Standards Is Not AI Governancecited by 9
  8. 2025In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AIcited by 7
  9. 2025Red Teaming Contemporary AI Models: Insights from Spanish and Basque Perspectivescited by 6
  10. 2025Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Expertscited by 5
  11. 2025Red Teaming AI Red Teamingcited by 3
  12. 2025From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teamingcited by 2
16 more
  1. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  2. 2025Longitudinal Monitoring of LLM Content Moderation of Social Issuescited by 2
  3. 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
  4. 2026Beyond Abstract Compliance: Operationalising trust in AI as a moral relationshipcited by 1
  5. 2026Buy versus Build an LLM: A Decision Framework for Governmentscited by 1
  6. 2025Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Centercited by 0
  7. 2025PolicyPad: Collaborative Prototyping of LLM Policiescited by 0
  8. 2026Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimizationcited by 0
  9. 2026Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluationcited by 0
  10. 2026Red-Teaming Claude Opus and ChatGPT-based Security Advisors for Trusted Execution Environmentscited by 0
  11. 2026ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward Systemcited by 0
  12. 2026AVISE: Framework for Evaluating the Security of AI Systemscited by 0
  13. 2026FlashRT: Towards Computationally and Memory Efficient Red-Teaming for Prompt Injection and Knowledge Corruptioncited by 0
  14. 2026Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolutioncited by 0
  15. 2026Position: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safetycited by 0
  16. 2026A Threshold Exceedance Framework for CBRN Uplift Evaluation in Frontier Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Ask What Your Country Can Do For You: Towards a Public Red Teaming Model
  2. 2024The Human Factor in AI Red Teaming: Perspectives from Social and Collaborative Computing
  3. 2024AI red-teaming is a sociotechnical problem: on values, labor, and harms
  4. 2025Lessons From Red Teaming 100 Generative AI Products
  5. 2025A Red Teaming Roadmap Towards System-Level Safety
  6. 2025Effective Automation to Support the Human Infrastructure in AI Red Teaming
  7. 2024A Safe Harbor for AI Evaluation and Red Teaming
  8. 2026A Translational Note on AI Safety Evaluation