# PyRIT: A Framework for Security Risk Identification and Red Teaming in Generative AI System

arXiv: https://arxiv.org/abs/2410.02828 · HTML: https://mlsanity.com/paper/2410.02828/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 15 of 15)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack](https://mlsanity.com/paper/2404.01833/) (2024)
- [InterpretML: A Unified Framework for Machine Learning Interpretability](https://mlsanity.com/paper/1909.09223/) (2019)
- [Generative AI](https://mlsanity.com/paper/2309.07930/) (2023)
- [CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models](https://mlsanity.com/paper/2402.16717/) (2024)
- [Fairlearn: Assessing and Improving Fairness of AI Systems](https://mlsanity.com/paper/2303.16626/) (2023)
- [Datasets for Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2402.18041/) (2024)
- [garak: A Framework for Security Probing Large Language Models](https://mlsanity.com/paper/2406.11036/) (2024)

## Built on by (most cited first, 30 of 31)

- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [DoomArena: A framework for Testing AI Agents Against Evolving Security Threats](https://mlsanity.com/paper/2504.14064/) (2025)
- [Lessons From Red Teaming 100 Generative AI Products](https://mlsanity.com/paper/2501.07238/) (2025)
- [From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?](https://mlsanity.com/paper/2506.08295/) (2025)
- [OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs](https://mlsanity.com/paper/2601.01592/) (2026)
- [SoSBench: Benchmarking Safety Alignment on Six Scientific Domains](https://mlsanity.com/paper/2505.21605/) (2025)
- [Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models](https://mlsanity.com/paper/2503.01742/) (2025)
- [Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis](https://mlsanity.com/paper/2410.16527/) (2024)
- [SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses](https://mlsanity.com/paper/2510.15476/) (2025)
- [Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO](https://mlsanity.com/paper/2606.09701/) (2026)
- [aiXamine: Simplified LLM Safety and Security](https://mlsanity.com/paper/2504.14985/) (2025)
- [Exploiting Web Search Tools of AI Agents for Data Exfiltration](https://mlsanity.com/paper/2510.09093/) (2025)
- [K2-V2: A 360-Open, Reasoning-Enhanced LLM](https://mlsanity.com/paper/2512.06201/) (2025)
- [Adaptive Instruction Composition for Automated LLM Red-Teaming](https://mlsanity.com/paper/2604.21159/) (2026)
- [SoK: Robustness in Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2605.05058/) (2026)
- [AgenticVM: Agentic AI for Adaptive Software Vulnerability Management](https://mlsanity.com/paper/2605.01739/) (2026)
- [Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming](https://mlsanity.com/paper/2606.31227/) (2026)
- [MAD-MAX: Modular And Diverse Malicious Attack MiXtures for Automated LLM Red Teaming](https://mlsanity.com/paper/2503.06253/) (2025)
- [AJAR: Adaptive Jailbreak Architecture for Red-teaming](https://mlsanity.com/paper/2601.10971/) (2026)
- [When Scanners Lie: Evaluator Instability in LLM Red-Teaming](https://mlsanity.com/paper/2603.14633/) (2026)
- [Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours](https://mlsanity.com/paper/2605.04019/) (2026)
- [Auction-Based Regulation for Artificial Intelligence](https://mlsanity.com/paper/2410.01871/) (2024)
- [BlackIce: A Containerized Red Teaming Toolkit for AI Security Testing](https://mlsanity.com/paper/2510.11823/) (2025)
- [MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Models](https://mlsanity.com/paper/2603.02482/) (2026)
- [AVISE: Framework for Evaluating the Security of AI Systems](https://mlsanity.com/paper/2604.20833/) (2026)
- [Beyond Pass/Fail: Using Process Mining to Understand How LLMs Resist (and Fail) Red Team Attacks](https://mlsanity.com/paper/2606.07833/) (2026)
- [Jailbreaking for the Average Jane: Choosing Optimal Jailbreaks via Bandit Algorithms for Automatically Enhanced Queries](https://mlsanity.com/paper/2606.26936/) (2026)
- [Granite.Trust Policy Tools: Shareable, Actionable Policies for Generative AI Applications](https://mlsanity.com/paper/2608.23870/) (2026)
- [EvoHarmBench: Breaking Content Moderation with Iterative Human-Like Evasion](https://mlsanity.com/paper/2608.27844/) (2026)
- [ASCII Attack: Recontextualising Harmful Requests as Artistic Critique in Large Language Models](https://mlsanity.com/paper/2609.02215/) (2026)

## Abstract (arXiv)

Generative Artificial Intelligence (GenAI) is becoming ubiquitous in our daily lives. The increase in computational power and data availability has led to a proliferation of both single- and multi-modal models. As the GenAI ecosystem matures, the need for extensible and model-agnostic risk identification frameworks is growing. To meet this need, we introduce the Python Risk Identification Toolkit (PyRIT), an open-source framework designed to enhance red teaming efforts in GenAI systems. PyRIT is a model- and platform-agnostic tool that enables red teamers to probe for and identify novel harms, risks, and jailbreaks in multimodal generative AI models. Its composable architecture facilitates the reuse of core building blocks and allows for extensibility to future models and modalities. This paper details the challenges specific to red teaming generative AI systems, the development and features of PyRIT, and its practical applications in real-world scenarios.