# Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models

arXiv: https://arxiv.org/abs/2506.07121 · HTML: https://mlsanity.com/paper/2506.07121/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 57 of its 69 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 30 of 57)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation](https://mlsanity.com/paper/2106.04399/) (2021)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts](https://mlsanity.com/paper/2402.16822/) (2024)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs](https://mlsanity.com/paper/2410.05295/) (2024)
- [ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models](https://mlsanity.com/paper/2310.10505/) (2023)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models](https://mlsanity.com/paper/2402.13457/) (2024)

## Built on by (most cited first, 9 of 9)

- [Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts](https://mlsanity.com/paper/2602.03545/) (2026)
- [AgenticRed: Evolving Agentic Systems for Red-Teaming](https://mlsanity.com/paper/2601.13518/) (2026)
- [TEAR: Temporal-aware Automated Red-teaming for Text-to-Video Models](https://mlsanity.com/paper/2511.21145/) (2025)
- [Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming](https://mlsanity.com/paper/2607.11698/) (2026)
- [Soft Quality-Diversity Optimization](https://mlsanity.com/paper/2512.00810/) (2025)
- [Belief-Guided Inference Control for Large Language Model Services via Verifiable Observations](https://mlsanity.com/paper/2604.27536/) (2026)
- [Quality-Diversity Optimization as Multi-Objective Optimization](https://mlsanity.com/paper/2602.00478/) (2026)
- [Quality-Diversity Stress Tests for Process Reward Models:What Archive Coverage Can and Cannot Certify](https://mlsanity.com/paper/2608.08008/) (2026)
- [X-AgentRed: Red-Teaming Personal Agents in Evolving Multi-Service Ecosystems](https://mlsanity.com/paper/2610.13886/) (2026)

## Abstract (arXiv)

Ensuring the safety and robustness of large language models (LLMs) is a fundamental challenge and a critical prerequisite for the responsible deployment of artificial intelligence. Red-teaming, a systematic framework to identify adversarial prompts that elicit harmful responses from target LLMs, has emerged as a crucial safety evaluation paradigm. Within this framework, the diversity of adversarial prompts is critical for a comprehensive safety assessment. However, previous red-teaming approaches often pursue diversity through simplistic metrics such as word frequency or sentence embedding similarity, which may not capture meaningful variation in attack strategies. In addition, the common practice of training a single attacker model restricts coverage across all potential attack styles and risk categories. This paper introduces Quality-Diversity Red-Teaming (QDRT), a new framework designed to address these limitations. QDRT achieves goal-driven diversity through behavior-conditioned training and implements a behavioral replay buffer in an open-ended manner. In addition, it trains multiple specialized attackers capable of generating high-quality attacks across diverse styles and risk categories. Our empirical evaluation demonstrates that QDRT generates attacks that are both more diverse and more effective against a wide range of target LLMs, including open-source models textttGPT-2, textttLlama-3, textttGemma-2, textttQwen2.5, and commercial models textttGPT-4.1 and textttGPT-5-Chat. This work advances the field of LLM safety by providing a systematic and effective approach to automated red-teaming, ultimately supporting the responsible deployment of LLMs. Our code is available at https://github.com/lamda-bbo/QDRT.