# Ruby Teaming: Improving Quality Diversity Search with Memory for Automated Red Teaming

arXiv: https://arxiv.org/abs/2406.11654 · HTML: https://mlsanity.com/paper/2406.11654/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 10 of its 23 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 10 of 10)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts](https://mlsanity.com/paper/2402.16822/) (2024)
- [FLIRT: Feedback Loop In-context Red Teaming](https://mlsanity.com/paper/2308.04265/) (2023)
- [Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biases](https://mlsanity.com/paper/2310.14303/) (2023)

## Built on by (most cited first, 10 of 10)

- [RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Search](https://mlsanity.com/paper/2504.15047/) (2025)
- [Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models](https://mlsanity.com/paper/2506.07121/) (2025)
- [PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teaming](https://mlsanity.com/paper/2509.03728/) (2025)
- [Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique](https://mlsanity.com/paper/2408.10701/) (2024)
- [Active Attacks: Red-teaming LLMs via Adaptive Environments](https://mlsanity.com/paper/2509.21947/) (2025)
- [Exploring Empty Spaces: Human-in-the-Loop Data Augmentation](https://mlsanity.com/paper/2410.01088/) (2024)
- [Towards Effective Discrimination Testing for Generative AI](https://mlsanity.com/paper/2412.21052/) (2024)
- [Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balance](https://mlsanity.com/paper/2605.00553/) (2026)
- [PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AI](https://mlsanity.com/paper/2605.05682/) (2026)
- [Generating Attacks for LLMs with GFlowNets](https://mlsanity.com/paper/2608.10171/) (2026)

## Abstract (arXiv)

We propose Ruby Teaming, a method that improves on Rainbow Teaming by including a memory cache as its third dimension. The memory dimension provides cues to the mutator to yield better-quality prompts, both in terms of attack success rate (ASR) and quality diversity. The prompt archive generated by Ruby Teaming has an ASR of 74%, which is 20% higher than the baseline. In terms of quality diversity, Ruby Teaming outperforms Rainbow Teaming by 6% and 3% on Shannon's Evenness Index (SEI) and Simpson's Diversity Index (SDI), respectively.