mlsanityData to 7 Oct 2026

Curiosity-driven Red-teaming for Large Language Models

arXiv 2402.19464 ↗First posted 29 Feb 2024cs.LG cs.AI cs.CLcited by 83

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large language models (LLMs) hold great potential for many natural language applications but risk generating incorrect or toxic content. To probe when an LLM generates unwanted content, the current paradigm is to recruit a textitred team of human testers to design input prompts (i.e., test cases) that elicit undesirable responses from LLMs. However, relying solely on human testers is expensive and time-consuming. Recent works automate red teaming by training a separate red team LLM with reinforcement learning (RL) to generate test cases that maximize the chance of eliciting undesirable responses from the target LLM. However, current RL methods are only able to generate a small number of effective test cases resulting in a low coverage of the span of prompts that elicit undesirable responses from the target LLM. To overcome this limitation, we draw a connection between the problem of increasing the coverage of generated test cases and the well-studied approach of curiosity-driven exploration that optimizes for novelty. Our method of curiosity-driven red teaming (CRT) achieves greater coverage of test cases while mantaining or increasing their effectiveness compared to existing methods. Our method, CRT successfully provokes toxic responses from LLaMA2 model that has been heavily fine-tuned using human preferences to avoid toxic outputs. Code is available at urlhttps://github.com/Improbable-AI/curiosityredteam

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 34 of its 50 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

34 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch15 Jun
  2. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimization27 May
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  4. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
18 more
  1. 2018Exploration by Random Network Distillationcited by 966
  2. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  3. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  4. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  5. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  6. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  7. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  8. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  9. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  10. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  11. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  12. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  13. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
  14. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  15. 2019Jointly Measuring Diversity and Quality in Text Generation Modelscited by 61
  16. 2023Learning to Generate Better Than Your LLMcited by 49
  17. 2022Redeeming Intrinsic Rewards via Constrained Optimizationcited by 24
  18. 2023Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editorscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 40; 2026: 17.

83 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  3. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  4. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 46
  5. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  6. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  7. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  8. 2024ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Userscited by 32
  9. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
  10. 2024Protecting Your LLMs with Information Bottleneckcited by 26
  11. 2024Adaptive teachers for amortized samplerscited by 23
  12. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
38 more
  1. 2024Locking Down the Finetuned LLMs Safetycited by 22
  2. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  3. 2025Searching for Privacy Risks in LLM Agents via Simulationcited by 19
  4. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
  5. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  6. 2024HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Modelscited by 17
  7. 2024Superficial Safety Alignment Hypothesiscited by 14
  8. 2024Toward Optimal LLM Alignments Using Two-Player Gamescited by 13
  9. 2024Generating Out-Of-Distribution Scenarios Using Language Modelscited by 10
  10. 2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10
  11. 2025Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderationcited by 9
  12. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signalscited by 9
  13. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 8
  14. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  15. 2024SEAS: Self-Evolving Adversarial Safety Optimization for Large Language Modelscited by 7
  16. 2024Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Explorationcited by 7
  17. 2025RedDebate: Safer Responses Through Multi-Agent Red Teaming Debatescited by 7
  18. 2025Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Trainingcited by 7
  19. 2024Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teamingcited by 6
  20. 2025CALM: Curiosity-Driven Auditing for Large Language Modelscited by 6
  21. 2024RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fieldscited by 5
  22. 2025EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistantscited by 5
  23. 2025DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modelingcited by 5
  24. 2025Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimizationcited by 5
  25. 2025Active Attacks: Red-teaming LLMs via Adaptive Environmentscited by 5
  26. 2024Towards Action Hijacking of Large Language Model-based Agentcited by 4
  27. 2025Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraintscited by 4
  28. 2025GenBreak: Red Teaming Text-to-Image Generators Using Large Language Modelscited by 4
  29. 2025Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalitiescited by 4
  30. 2025Geometric Red-Teaming for Robotic Manipulationcited by 4
  31. 2025Probing Knowledge Holes in Unlearned LLMscited by 4
  32. 2025Tailored Primitive Initialization is the Secret Key to Reinforcement Learningcited by 4
  33. 2025Eliciting Behaviors in Multi-Turn Conversationscited by 4
  34. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Modelscited by 4
  35. 2024ROER: Regularized Optimal Experience Replaycited by 3
  36. 2024LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Secondscited by 3
  37. 2025Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Expertscited by 3
  38. 2025TEAR: Temporal-aware Automated Red-teaming for Text-to-Video Modelscited by 3

50 most cited of 83 All 83 (JSON)