mlsanityData to 14 Oct 2026

PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI System

arXiv 2410.02828 ↗First posted 1 Oct 2024cs.CR cs.AI cs.CLcited by 31

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Generative Artificial Intelligence (GenAI) is becoming ubiquitous in our daily lives. The increase in computational power and data availability has led to a proliferation of both single- and multi-modal models. As the GenAI ecosystem matures, the need for extensible and model-agnostic risk identification frameworks is growing. To meet this need, we introduce the Python Risk Identification Toolkit (PyRIT), an open-source framework designed to enhance red teaming efforts in GenAI systems. PyRIT is a model- and platform-agnostic tool that enables red teamers to probe for and identify novel harms, risks, and jailbreaks in multimodal generative AI models. Its composable architecture facilitates the reuse of core building blocks and allows for extensibility to future models and modalities. This paper details the challenges specific to red teaming generative AI systems, the development and features of PyRIT, and its practical applications in real-world scenarios.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 15 of its 27 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

15 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024garak: A Framework for Security Probing Large Language Models16 Jun

Other references · 14

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  5. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  6. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  8. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  9. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  10. 2019InterpretML: A Unified Framework for Machine Learning Interpretabilitycited by 246
  11. 2023Generative AIcited by 120
  12. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 76
2 more
  1. 2023Fairlearn: Assessing and Improving Fairness of AI Systemscited by 74
  2. 2024Datasets for Large Language Models: A Comprehensive Surveycited by 71

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 17.

31 papers.

Sort:

By area:

  1. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
  2. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 32
  3. 2025Lessons From Red Teaming 100 Generative AI Productscited by 23
  4. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 23
  5. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMscited by 21
  6. 2025SoSBench: Benchmarking Safety Alignment on Six Scientific Domainscited by 12
  7. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  8. 2024Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysiscited by 10
  9. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
  10. 2026Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPOcited by 5
  11. 2025aiXamine: Simplified LLM Safety and Securitycited by 4
  12. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltrationcited by 3
19 more
  1. 2025K2-V2: A 360-Open, Reasoning-Enhanced LLMcited by 3
  2. 2026Adaptive Instruction Composition for Automated LLM Red-Teamingcited by 3
  3. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 3
  4. 2026AgenticVM: Agentic AI for Adaptive Software Vulnerability Managementcited by 2
  5. 2026Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teamingcited by 2
  6. 2025MAD-MAX: Modular And Diverse Malicious Attack MiXtures for Automated LLM Red Teamingcited by 1
  7. 2026AJAR: Adaptive Jailbreak Architecture for Red-teamingcited by 1
  8. 2026When Scanners Lie: Evaluator Instability in LLM Red-Teamingcited by 1
  9. 2026Redefining AI Red Teaming in the Agentic Era: From Weeks to Hourscited by 1
  10. 2024Auction-Based Regulation for Artificial Intelligencecited by 0
  11. 2025BlackIce: A Containerized Red Teaming Toolkit for AI Security Testingcited by 0
  12. 2026MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Modelscited by 0
  13. 2026AVISE: Framework for Evaluating the Security of AI Systemscited by 0
  14. 2026Beyond Pass/Fail: Using Process Mining to Understand How LLMs Resist (and Fail) Red Team Attackscited by 0
  15. 2026Jailbreaking for the Average Jane: Choosing Optimal Jailbreaks via Bandit Algorithms for Automatically Enhanced Queriescited by 0
  16. 2026Granite.Trust Policy Tools: Shareable, Actionable Policies for Generative AI Applicationscited by 0
  17. 2026EvoHarmBench: Breaking Content Moderation with Iterative Human-Like Evasioncited by 0
  18. 2026ASCII Attack: Recontextualising Harmful Requests as Artistic Critique in Large Language Modelscited by 0
  19. 2026Defensive Sufficiency in a Stackelberg Model of AI Securitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation
  2. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?
  3. 2025OpenAI's Approach to External Red Teaming for AI Models and Systems
  4. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI
  5. 2025Red Teaming AI Red Teaming
  6. 2024A Safe Harbor for AI Evaluation and Red Teaming
  7. 2026Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
  8. 2025Demo: ViolentUTF as An Accessible Platform for Generative AI Red Teaming