# AIRTBench: Measuring Autonomous AI Red Teaming Capabilities in Language Models

arXiv: https://arxiv.org/abs/2506.14682 · HTML: https://mlsanity.com/paper/2506.14682/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 18 of 18)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024)
- [NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security](https://mlsanity.com/paper/2406.05590/) (2024)
- [EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities](https://mlsanity.com/paper/2409.16165/) (2024)
- [LLM4Decompile: Decompiling Binary Code with Large Language Models](https://mlsanity.com/paper/2403.05286/) (2024)
- [An Empirical Evaluation of LLMs for Solving Offensive Security Challenges](https://mlsanity.com/paper/2402.11814/) (2024)
- [AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defenses](https://mlsanity.com/paper/2503.01811/) (2025)
- [Optimizing Large Language Model Hyperparameters for Code Generation](https://mlsanity.com/paper/2408.10577/) (2024)
- [Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for Large Language Models Aligned with Human Cognitive Principles](https://mlsanity.com/paper/2406.12644/) (2024)
- [The Automation Advantage in AI Red Teaming](https://mlsanity.com/paper/2504.19855/) (2025)

## Built on by (most cited first, 7 of 7)

- [PoCo: Agentic Proof-of-Concept Exploit Generation for Smart Contracts](https://mlsanity.com/paper/2511.02780/) (2025)
- [PentestJudge: Judging Agent Behavior Against Operational Requirements](https://mlsanity.com/paper/2508.02921/) (2025)
- [Evaluating LLM Generated Detection Rules in Cybersecurity](https://mlsanity.com/paper/2509.16749/) (2025)
- [Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security](https://mlsanity.com/paper/2507.19399/) (2025)
- [Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours](https://mlsanity.com/paper/2605.04019/) (2026)
- [Terminal Agents: A Survey of AI Agents in Command-Line Environments](https://mlsanity.com/paper/2608.20485/) (2026)
- [A Translational Note on AI Safety Evaluation](https://mlsanity.com/paper/2609.06573/) (2026)

## Abstract (arXiv)

We introduce AIRTBench, an AI red teaming benchmark for evaluating language models' ability to autonomously discover and exploit Artificial Intelligence and Machine Learning (AI/ML) security vulnerabilities. The benchmark consists of 70 realistic black-box capture-the-flag (CTF) challenges from the Crucible challenge environment on the Dreadnode platform, requiring models to write python code to interact with and compromise AI systems. Claude-3.7-Sonnet emerged as the clear leader, solving 43 challenges (61% of the total suite, 46.9% overall success rate), with Gemini-2.5-Pro following at 39 challenges (56%, 34.3% overall), GPT-4.5-Preview at 34 challenges (49%, 36.9% overall), and DeepSeek R1 at 29 challenges (41%, 26.9% overall). Our evaluations show frontier models excel at prompt injection attacks (averaging 49% success rates) but struggle with system exploitation and model inversion challenges (below 26%, even for the best performers). Frontier models are far outpacing open-source alternatives, with the best truly open-source model (Llama-4-17B) solving 7 challenges (10%, 1.0% overall), though demonstrating specialized capabilities on certain hard challenges. Compared to human security researchers, large language models (LLMs) solve challenges with remarkable efficiency completing in minutes what typically takes humans hours or days-with efficiency advantages of over 5,000x on hard challenges. Our contribution fills a critical gap in the evaluation landscape, providing the first comprehensive benchmark specifically designed to measure and track progress in autonomous AI red teaming capabilities.