mlsanityData to 7 Oct 2026

Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction

arXiv 2409.16783 ↗First posted 25 Sep 2024cs.CL cs.AI cs.CRcited by 25

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Automated red teaming is an effective method for identifying misaligned behaviors in large language models (LLMs). Existing approaches, however, often focus primarily on improving attack success rates while overlooking the need for comprehensive test case coverage. Additionally, most of these methods are limited to single-turn red teaming, failing to capture the multi-turn dynamics of real-world human-machine interactions. To overcome these limitations, we propose HARM (Holistic Automated Red teaMing), which scales up the diversity of test cases using a top-down approach based on an extensible, fine-grained risk taxonomy. Our method also leverages a novel fine-tuning strategy and reinforcement learning techniques to facilitate multi-turn adversarial probing in a human-like manner. Experimental results demonstrate that our framework enables a more systematic understanding of model vulnerabilities and offers more targeted guidance for the alignment process.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 29 of its 44 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

29 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  2. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
  3. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment18 Aug
  4. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch15 Jun
  5. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimization27 May
  6. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  12. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
11 more
  1. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  2. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  3. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  4. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  5. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  6. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  7. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  8. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  9. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  10. 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 103
  11. 2023ROBBIE: Robust Bias Evaluation of Large Generative Language Modelscited by 53

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 11.

25 papers.

Sort:

By area:

  1. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 31
  2. 2025LLM Agents Should Employ Security Principlescited by 28
  3. 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attackscited by 23
  4. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Modelscited by 19
  5. 2025Large Reasoning Models Are Autonomous Jailbreak Agentscited by 12
  6. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  7. 2025TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewardscited by 11
  8. 2025RedCoder: Automated Multi-Turn Red Teaming for Code LLMscited by 8
  9. 2025Adversarial Preference Learning for Robust LLM Alignmentcited by 7
  10. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  11. 2026Knowledge-Driven Multi-Turn Jailbreaking on Large Language Modelscited by 6
  12. 2025A Review of Incorporating Psychological Theories in LLMscited by 5
13 more
  1. 2025Automatic LLM Red Teamingcited by 5
  2. 2025Eliciting Behaviors in Multi-Turn Conversationscited by 4
  3. 2026Alignment Verifiability in Large Language Models: Normative Indistinguishability under Behavioral Evaluationcited by 2
  4. 2026When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraintscited by 2
  5. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  6. 2026JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Modelscited by 1
  7. 2026Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignmentcited by 1
  8. 2026CIRCLE: A Framework for Evaluating AI from a Real-World Lenscited by 1
  9. 2026CRaFT: Circuit-Guided Refusal Feature Selection via Cross-Layer Transcoderscited by 1
  10. 2025Red-teaming Activation Probes using Prompted LLMscited by 0
  11. 2026How Few-shot Demonstrations Affect Prompt-based Defenses Against LLM Jailbreak Attackscited by 0
  12. 2026NodeSynth: Socially Aligned Synthetic Data for AI Evaluationcited by 0
  13. 2026Robust Critics: Defending LLMs Against Multi-Turn Attackscited by 0