mlsanityData to 14 Oct 2026

Automatic LLM Red Teaming

arXiv 2508.04451 ↗First posted 6 Aug 2025cs.LG cs.AIcited by 5

On the map: LLM safety and security → Prompt injection and agent security

also in 1 more
Abstract (arXiv)

Red teaming is essential for securing Large Language Models, yet current automated methods remain limited by templates and single-turn attacks. To simulate the complex, interactive nature of real-world adversarial attacks, we introduce a novel red teaming paradigm designed to maximize expected cumulative harm through strategic interaction. By formalizing red teaming as a Markov Decision Process in a hierarchical reinforcement learning framework, we navigate the challenges of sparse rewards and long-horizon planning. Our generative agent learns diverse, multi-turn attacks using a token-level harm reward, consistently uncovering vulnerabilities that bypass baselines. This approach achieves a new state of the art and reframes LLM red teaming as a principled, trajectory-based process.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 21 of its 28 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

21 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction25 Sep
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  3. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
  4. 2023Evolving Diverse Red-team Language Models in Multi-round Multi-agent Games30 Sep

Other references · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  8. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  9. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  10. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  11. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  12. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 96
5 more
  1. 2024Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLMcited by 77
  2. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  3. 2025Tempest: Autonomous Multi-Turn Jailbreaking of Large Language Models with Tree Searchcited by 17
  4. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  5. 2023Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attackcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guardcited by 13
  2. 2026Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planningcited by 4
  3. 2026QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarkscited by 1
  4. 2026Emergent Alignmentcited by 0
  5. 2026Robust Critics: Defending LLMs Against Multi-Turn Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations
  2. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuning
  3. 2025RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models
  4. 2025RedCoder: Automated Multi-Turn Red Teaming for Code LLMs
  5. 2026A Red Teaming Framework for Evaluating Robustness of AI-enabled Security Orchestration, Automation, and Response Systems
  6. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Tester
  7. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Models
  8. 2025RedTWIZ: Diverse LLM Red Teaming via Adaptive Attack Planning