# AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration

arXiv: https://arxiv.org/abs/2503.15754 · HTML: https://mlsanity.com/paper/2503.15754/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 37 of its 51 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 30 of 37)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [Improving Alignment and Robustness with Circuit Breakers](https://mlsanity.com/paper/2406.04313/) (2024)
- [SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models](https://mlsanity.com/paper/2402.05044/) (2024)
- [WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models](https://mlsanity.com/paper/2406.18510/) (2024)
- [Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts](https://mlsanity.com/paper/2402.16822/) (2024)
- [AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs](https://mlsanity.com/paper/2410.05295/) (2024)
- [MART: Improving LLM Safety with Multi-round Automatic Red-Teaming](https://mlsanity.com/paper/2311.07689/) (2023)
- [Sociotechnical Safety Evaluation of Generative AI Systems](https://mlsanity.com/paper/2310.11986/) (2023)
- [Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2401.17263/) (2024)
- [LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet](https://mlsanity.com/paper/2408.15221/) (2024)
- [Open Sesame! Universal Black Box Jailbreaking of Large Language Models](https://mlsanity.com/paper/2309.01446/) (2023)

## Built on by (most cited first, 30 of 43)

- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [Security Concerns for Large Language Models: A Survey](https://mlsanity.com/paper/2505.18889/) (2025)
- [OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs](https://mlsanity.com/paper/2601.01592/) (2026)
- [AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models](https://mlsanity.com/paper/2509.26100/) (2025)
- [GeneBreaker: Jailbreak Attacks against DNA Language Models with Pathogenicity Guidance](https://mlsanity.com/paper/2505.23839/) (2025)
- [Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMs](https://mlsanity.com/paper/2511.12710/) (2025)
- [The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search](https://mlsanity.com/paper/2512.01353/) (2025)
- [SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Models](https://mlsanity.com/paper/2509.03487/) (2025)
- [ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attacks](https://mlsanity.com/paper/2510.02677/) (2025)
- [SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning](https://mlsanity.com/paper/2510.26037/) (2025)
- [AgenticRed: Evolving Agentic Systems for Red-Teaming](https://mlsanity.com/paper/2601.13518/) (2026)
- [When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?](https://mlsanity.com/paper/2510.17862/) (2025)
- [PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploits](https://mlsanity.com/paper/2510.17947/) (2025)
- [TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration](https://mlsanity.com/paper/2603.22882/) (2026)
- [Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judges](https://mlsanity.com/paper/2511.01375/) (2025)
- [Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agents](https://mlsanity.com/paper/2604.05549/) (2026)
- [AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Models](https://mlsanity.com/paper/2604.22871/) (2026)
- [Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems](https://mlsanity.com/paper/2605.11891/) (2026)
- [ADR: An Agentic Detection System for Enterprise Agentic AI Security](https://mlsanity.com/paper/2605.17380/) (2026)
- [PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections](https://mlsanity.com/paper/2606.12737/) (2026)
- [SCARCE: Scalable Cascade Analysis for Rare-event Characterisation via Embeddings](https://mlsanity.com/paper/2606.29623/) (2026)
- [Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming](https://mlsanity.com/paper/2607.11698/) (2026)
- [ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents](https://mlsanity.com/paper/2608.11878/) (2026)
- [Red-teaming Activation Probes using Prompted LLMs](https://mlsanity.com/paper/2511.00554/) (2025)
- [NAAMSE: Framework for Evolutionary Security Evaluation of Agents](https://mlsanity.com/paper/2602.07391/) (2026)
- [Large Language Models in the Abuse Detection Pipeline](https://mlsanity.com/paper/2604.00323/) (2026)
- [Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours](https://mlsanity.com/paper/2605.04019/) (2026)
- [JailbreakOPT: Tool-Assisted Iterative Jailbreak Prompt Optimization](https://mlsanity.com/paper/2606.11425/) (2026)
- [PolicyPad: Collaborative Prototyping of LLM Policies](https://mlsanity.com/paper/2509.19680/) (2025)
- [Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testing](https://mlsanity.com/paper/2604.20704/) (2026)

## Abstract (arXiv)

As large language models (LLMs) become increasingly capable, security and safety evaluation are crucial. While current red teaming approaches have made strides in assessing LLM vulnerabilities, they often rely heavily on human input and lack comprehensive coverage of emerging attack vectors. This paper introduces AutoRedTeamer, a novel framework for fully automated, end-to-end red teaming against LLMs. AutoRedTeamer combines a multi-agent architecture with a memory-guided attack selection mechanism to enable continuous discovery and integration of new attack vectors. The dual-agent framework consists of a red teaming agent that can operate from high-level risk categories alone to generate and execute test cases and a strategy proposer agent that autonomously discovers and implements new attacks by analyzing recent research. This modular design allows AutoRedTeamer to adapt to emerging threats while maintaining strong performance on existing attack vectors. We demonstrate AutoRedTeamer's effectiveness across diverse evaluation settings, achieving 20% higher attack success rates on HarmBench against Llama-3.1-70B while reducing computational costs by 46% compared to existing approaches. AutoRedTeamer also matches the diversity of human-curated benchmarks in generating test cases, providing a comprehensive, scalable, and continuously evolving framework for evaluating the security of AI systems.