mlsanityData to 7 Oct 2026

Ruby Teaming: Improving Quality Diversity Search with Memory for Automated Red Teaming

arXiv 2406.11654 ↗First posted 17 Jun 2024cs.CLcited by 10

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

We propose Ruby Teaming, a method that improves on Rainbow Teaming by including a memory cache as its third dimension. The memory dimension provides cues to the mutator to yield better-quality prompts, both in terms of attack success rate (ASR) and quality diversity. The prompt archive generated by Ruby Teaming has an ASR of 74%, which is 20% higher than the baseline. In terms of quality diversity, Ruby Teaming outperforms Rainbow Teaming by 6% and 3% on Shannon's Evenness Index (SEI) and Simpson's Diversity Index (SDI), respectively.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 10 of its 23 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

10 of 23 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  2. 2023Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biases22 Oct
  3. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 7

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2015Illuminating search spaces by mapping elitescited by 406
  7. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 3.

10 papers.

  1. 2025RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Searchcited by 10
  2. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 8
  3. 2025PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teamingcited by 8
  4. 2024Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Techniquecited by 7
  5. 2024Exploring Empty Spaces: Human-in-the-Loop Data Augmentationcited by 6
  6. 2024Towards Effective Discrimination Testing for Generative AIcited by 5
  7. 2025Active Attacks: Red-teaming LLMs via Adaptive Environmentscited by 5
  8. 2026Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balancecited by 0
  9. 2026PersonaTeaming: Supporting Persona-Driven Red-Teaming for Generative AIcited by 0
  10. 2026Generating Attacks for LLMs with GFlowNetscited by 0