mlsanityData to 14 Oct 2026

Be a Multitude to Itself: A Prompt Evolution Framework for Red Teaming

arXiv 2502.16109 ↗First posted 22 Feb 2025cs.CLcited by 1

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) have gained increasing attention for their remarkable capacity, alongside concerns about safety arising from their potential to produce harmful content. Red teaming aims to find prompts that could elicit harmful responses from LLMs, and is essential to discover and mitigate safety risks before real-world deployment. However, manual red teaming is both time-consuming and expensive, rendering it unscalable. In this paper, we propose RTPE, a scalable evolution framework to evolve red teaming prompts across both breadth and depth dimensions, facilitating the automatic generation of numerous high-quality and diverse red teaming prompts. Specifically, in-breadth evolving employs a novel enhanced in-context learning method to create a multitude of quality prompts, whereas in-depth evolving applies customized transformation operations to enhance both content and form of prompts, thereby increasing diversity. Extensive experiments demonstrate that RTPE surpasses existing representative automatic red teaming methods on both attack success rate and diversity. In addition, based on 4,800 red teaming prompts created by RTPE, we further provide a systematic analysis of 8 representative LLMs across 8 sensitive topics.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 27 of its 41 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

27 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks16 Oct
  2. 2023ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models14 Oct
  3. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment18 Aug
  4. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked14 Aug

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  9. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  10. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  11. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  12. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
11 more
  1. 2023Are aligned neural networks adversarially aligned?cited by 315
  2. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  3. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 95
  4. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 85
  5. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 83
  6. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 68
  7. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
  8. 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectorscited by 51
  9. 2023Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attackscited by 34
  10. 2023Small Language Models Improve Giants by Rewriting Their Outputscited by 18
  11. 2023Large Language Models Meet Computer Vision: A Brief Surveycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Alignment Flywheel: A Governance-Centric Hybrid MAS for Architecture-Agnostic Safetycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuning
  2. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Models
  3. 2025RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Search
  4. 2026CART: Closed-Loop Adaptive Red Teaming for Large Language Models
  5. 2026REPRO: Reflective Evolutionary Prompt Optimizer for Automated LLM Red-Teaming
  6. 2025Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
  7. 2025RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models
  8. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models