mlsanityData to 7 Oct 2026

Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts

arXiv 2402.16822 ↗First posted 26 Feb 2024cs.CL cs.AI cs.LGcited by 189

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

As large language models (LLMs) become increasingly prevalent across many real-world applications, understanding and enhancing their robustness to adversarial attacks is of paramount importance. Existing methods for identifying adversarial prompts tend to focus on specific domains, lack diversity, or require extensive human annotations. To address these limitations, we present Rainbow Teaming, a novel black-box approach for producing a diverse collection of adversarial prompts. Rainbow Teaming casts adversarial prompt generation as a quality-diversity problem and uses open-ended search to generate prompts that are both effective and diverse. Focusing on the safety domain, we use Rainbow Teaming to target various state-of-the-art LLMs, including the Llama 2 and Llama 3 models. Our approach reveals hundreds of effective adversarial prompts, with an attack success rate exceeding 90% across all tested models. Furthermore, we demonstrate that prompts generated by Rainbow Teaming are highly transferable and that fine-tuning models with synthetic data generated by our method significantly enhances their safety without sacrificing general performance or helpfulness. We additionally explore the versatility of Rainbow Teaming by applying it to question answering and cybersecurity, showcasing its potential to drive robust open-ended self-improvement in a wide range of applications.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 42 of its 59 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

42 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
  2. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 40

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  12. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
28 more
  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  3. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  4. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  5. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  6. 2015Illuminating search spaces by mapping elitescited by 406
  7. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
  8. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  9. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  10. 2017Automated Curriculum Learning for Neural Networkscited by 251
  11. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  12. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  13. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  14. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  15. 2019Active Domain Randomizationcited by 161
  16. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  17. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  18. 2023Language Model Crossover: Variation through Few-Shot Promptingcited by 119
  19. 2017Quality and Diversity Optimization: A Unifying Modular Frameworkcited by 111
  20. 2021Replay-Guided Adversarial Environment Designcited by 108
  21. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  22. 2019Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptorscited by 57
  23. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  24. 2021On the Importance of Environments in Human-Robot Coordinationcited by 37
  25. 2023MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learningcited by 30
  26. 2023Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understandingcited by 28
  27. 2020Generating Automatic Curricula via Self-Supervised Active Domain Randomizationcited by 7
  28. 2024Multi-Agent Diagnostics for Robustness via Illuminated Diversitycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 42; 2025: 91; 2026: 56.

189 papers.

Sort:

By area:

  1. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  2. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  3. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  4. 2025Agentic Large Language Models, a surveycited by 105
  5. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  6. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 84
  7. 2024Best-of-N Jailbreakingcited by 71
  8. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  9. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  10. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
  11. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  12. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
38 more
  1. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
  2. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  3. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
  4. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  5. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  6. 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetitioncited by 29
  7. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  8. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
  9. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  10. 2024Investigating Implicit Bias in Large Language Models: A Large-Scale Study of Over 50 LLMscited by 25
  11. 2024GAVEL: Generating Games Via Evolution and Language Modelscited by 24
  12. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  13. 2024Embedding-based classifiers can detect prompt injection attackscited by 24
  14. 2026SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinementcited by 24
  15. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
  16. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  17. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  18. 2024Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Modelscited by 21
  19. 2024Aloe: A Family of Fine-tuned Open Healthcare LLMscited by 20
  20. 2025BingoGuard: LLM Content Moderation Tools with Risk Levelscited by 20
  21. 2025Searching for Privacy Risks in LLM Agents via Simulationcited by 19
  22. 2025Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effectscited by 18
  23. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  24. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  25. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  26. 2025Technological folie `a deux: Feedback Loops Between AI Chatbots and Mental Illnesscited by 14
  27. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  28. 2025Adversarial Reinforcement Learning for Large Language Model Agent Safetycited by 14
  29. 2024Toward Optimal LLM Alignments Using Two-Player Gamescited by 13
  30. 2024STAR: SocioTechnical Approach to Red Teaming Language Modelscited by 13
  31. 2025AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvementcited by 13
  32. 2025Evolution and The Knightian Blindspot of Machine Learningcited by 11
  33. 2025Adversarial Tokenizationcited by 11
  34. 2025CoP: Agentic Red-teaming for Large Language Models using Composition of Principlescited by 11
  35. 2025DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Modelscited by 11
  36. 2025TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewardscited by 11
  37. 2024Ruby Teaming: Improving Quality Diversity Search with Memory for Automated Red Teamingcited by 10
  38. 2024Agent Skill Acquisition for Large Language Models via CycleQDcited by 10

50 most cited of 189 All 189 (JSON)