mlsanityData to 14 Oct 2026

AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration

arXiv 2503.15754 ↗First posted 20 Mar 2025cs.CR cs.AIcited by 43

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

As large language models (LLMs) become increasingly capable, security and safety evaluation are crucial. While current red teaming approaches have made strides in assessing LLM vulnerabilities, they often rely heavily on human input and lack comprehensive coverage of emerging attack vectors. This paper introduces AutoRedTeamer, a novel framework for fully automated, end-to-end red teaming against LLMs. AutoRedTeamer combines a multi-agent architecture with a memory-guided attack selection mechanism to enable continuous discovery and integration of new attack vectors. The dual-agent framework consists of a red teaming agent that can operate from high-level risk categories alone to generate and execute test cases and a strategy proposer agent that autonomously discovers and implements new attacks by analyzing recent research. This modular design allows AutoRedTeamer to adapt to emerging threats while maintaining strong performance on existing attack vectors. We demonstrate AutoRedTeamer's effectiveness across diverse evaluation settings, achieving 20% higher attack success rates on HarmBench against Llama-3.1-70B while reducing computational costs by 46% compared to existing approaches. AutoRedTeamer also matches the diversity of human-curated benchmarks in generating test cases, providing a comprehensive, scalable, and continuously evolving framework for evaluating the security of AI systems.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 37 of its 51 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

37 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent23 Jul
  2. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  4. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  6. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  12. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
19 more
  1. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  2. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  3. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  4. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  5. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  6. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  7. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  8. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  9. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 199
  10. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  11. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  12. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  13. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 134
  14. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98
  15. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 79
  16. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 74
  17. 2023Managing extreme AI risks amid rapid progresscited by 48
  18. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 47
  19. 2024AI Risk Categorization Decoded (AIR 2024): From Government Regulations to Corporate Policiescited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 29.

43 papers.

Sort:

By area:

  1. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  2. 2025Security Concerns for Large Language Models: A Surveycited by 29
  3. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMscited by 21
  4. 2025AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Modelscited by 9
  5. 2025GeneBreaker: Jailbreak Attacks against DNA Language Models with Pathogenicity Guidancecited by 8
  6. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 8
  7. 2025The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Searchcited by 7
  8. 2025SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Modelscited by 5
  9. 2025ARMs: Adaptive Red-Teaming Agent against Multimodal Models with Plug-and-Play Attackscited by 5
  10. 2025SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoningcited by 5
  11. 2026AgenticRed: Evolving Agentic Systems for Red-Teamingcited by 5
  12. 2025When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?cited by 3
31 more
  1. 2025PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploitscited by 3
  2. 2026TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Explorationcited by 3
  3. 2025Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judgescited by 2
  4. 2026Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agentscited by 2
  5. 2026AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Modelscited by 2
  6. 2026Proteus: A Self-Evolving Red Team for Agent Skill Ecosystemscited by 2
  7. 2026ADR: An Agentic Detection System for Enterprise Agentic AI Securitycited by 2
  8. 2026PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injectionscited by 2
  9. 2026SCARCE: Scalable Cascade Analysis for Rare-event Characterisation via Embeddingscited by 2
  10. 2026Agent Hacks Agent: Autoresearch for Production-Agent Red-Teamingcited by 2
  11. 2026ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agentscited by 2
  12. 2025Red-teaming Activation Probes using Prompted LLMscited by 1
  13. 2026NAAMSE: Framework for Evolutionary Security Evaluation of Agentscited by 1
  14. 2026Large Language Models in the Abuse Detection Pipelinecited by 1
  15. 2026Redefining AI Red Teaming in the Agentic Era: From Weeks to Hourscited by 1
  16. 2026JailbreakOPT: Tool-Assisted Iterative Jailbreak Prompt Optimizationcited by 1
  17. 2025PolicyPad: Collaborative Prototyping of LLM Policiescited by 0
  18. 2026Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testingcited by 0
  19. 2026Model-Agnostic Lifelong LLM Safety via Externalized Attack-Defense Co-Evolutioncited by 0
  20. 2026REALM: A Unified Red-Teaming Benchmark for Physical-World VLMscited by 0
  21. 2026MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignmentcited by 0
  22. 2026Stress Testing Concept Erasure with Large Language Model Agentscited by 0
  23. 2026Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluationcited by 0
  24. 2026A Multimodal Automatic Redteaming Evaluation based on Atomic Jailbreak Strategy Decoupling and Combinationcited by 0
  25. 2026Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM Agentscited by 0
  26. 2026RedEvoAgent: Automatic Red-Teaming Agent with Experience-Driven Skill Evolutioncited by 0
  27. 2026SIR: Self-improving Red-teaming for Compute Use Agentscited by 0
  28. 2026Jailbreaking Text-to-Image Models Through Cracks: Navigating Heterogeneous Safety Filters via Multi-Agent Debatecited by 0
  29. 2026A Translational Note on AI Safety Evaluationcited by 0
  30. 2026ReSI: Recursive Safety Improvement toward Resistant and Resilient AIcited by 0
  31. 2026ATTCK-R: Knowledge-Aware Long-Horizon Jailbreaking for Protected LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools
  2. 2026JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills
  3. 2025Prompt Optimization and Evaluation for LLM Automated Red Teaming
  4. 2026REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems
  5. 2025Genesis: Evolving Attack Strategies for LLM Web Agent Red-Teaming
  6. 2026Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming
  7. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuning
  8. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models