mlsanityData to 7 Oct 2026

Lessons From Red Teaming 100 Generative AI Products

arXiv 2501.07238 ↗First posted 13 Jan 2025cs.AIcited by 22

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

In recent years, AI red teaming has emerged as a practice for probing the safety and security of generative AI systems. Due to the nascency of the field, there are many open questions about how red teaming operations should be conducted. Based on our experience red teaming over 100 generative AI products at Microsoft, we present our internal threat model ontology and eight main lessons we have learned: 1. Understand what the system can do and where it is applied 2. You don't have to compute gradients to break an AI system 3. AI red teaming is not safety benchmarking 4. Automation can help cover more of the risk landscape 5. The human element of AI red teaming is crucial 6. Responsible AI harms are pervasive but difficult to measure 7. LLMs amplify existing security risks and introduce new ones 8. The work of securing AI systems will never be complete By sharing these insights alongside case studies from our operations, we offer practical recommendations aimed at aligning red teaming efforts with real world risks. We also highlight aspects of AI red teaming that we believe are often misunderstood and discuss open questions for the field to consider.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 43 of its 56 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

43 of 56 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI System1 Oct
  2. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI23 Sep
  3. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)20 Jul
  4. 2024garak: A Framework for Security Probing Large Language Models16 Jun
  5. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  6. 2024Defending Against Indirect Prompt Injection Attacks With Spotlighting20 Mar
  7. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?29 Jan

Other references · 36

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  10. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  11. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  12. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
24 more
  1. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  2. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
  3. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  4. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  5. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  6. 2023AI Alignment: A Comprehensive Surveycited by 256
  7. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  8. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  9. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  10. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  11. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  12. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  13. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 121
  14. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  15. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  16. 2022The Fallacy of AI Functionalitycited by 88
  17. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  18. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  19. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 55
  20. 2024AI auditing: The Broken Bus on the Road to AI Accountabilitycited by 42
  21. 2024Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cyclecited by 32
  22. 2024What Was Your Prompt? A Remote Keylogging Attack on AI Assistantscited by 26
  23. 2024Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Datacited by 20
  24. 2024Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attackscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 8.

22 papers.

Sort:

By area:

  1. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  2. 2025Disentangling Reasoning and Knowledge in Medical Large Language Modelscited by 17
  3. 2025Real-World Gaps in AI Governance Researchcited by 5
  4. 2025Understanding and Mitigating Risks of Generative AI in Financial Servicescited by 4
  5. 2025Jailbreak Distillation: Renewable Safety Benchmarkingcited by 4
  6. 2025Red Teaming AI Red Teamingcited by 3
  7. 2025A Framework for Rapidly Developing and Deploying Protection Against Large Language Model Attackscited by 3
  8. 2025When Testing AI Tests Us: Safeguarding Mental Health on the Digital Frontlinescited by 2
  9. 2025The AI Model Risk Catalog: What Developers and Researchers Miss About Real-World AI Harmscited by 2
  10. 2025From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teamingcited by 2
  11. 2026Position: Stop Reactively Patching Your Model Every Time and Start Proactive Test-Driven AI Developmentcited by 2
  12. 2025Red Teaming Large Language Models for Healthcarecited by 1
10 more
  1. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  2. 2026A Systematic Review of Algorithmic Red Teaming Methodologies for Assurance and Security of AI Applicationscited by 1
  3. 2026Cascade: Composing Software-Hardware Attack Gadgets for Adversarial Threat Amplification in Compound AI Systemscited by 1
  4. 2026Redefining AI Red Teaming in the Agentic Era: From Weeks to Hourscited by 1
  5. 2025Responsible AI Technical Reportcited by 0
  6. 2025BlackIce: A Containerized Red Teaming Toolkit for AI Security Testingcited by 0
  7. 2026A Multi-Domain Red Teaming Framework for Safety, Robustness, and Fairness Evaluation of Medical Large Language Modelscited by 0
  8. 2026Decentralized Granular Access Control for Agentic AI Systems in Critical Infrastructurecited by 0
  9. 2026Beyond "I Can't Help With That": How Child Safety Experts Evaluate AI Chatbot Safetycited by 0
  10. 2026CART: Closed-Loop Adaptive Red Teaming for Large Language Modelscited by 0