mlsanityData to 7 Oct 2026

Learning diverse attacks on large language models for robust red-teaming and safety tuning

arXiv 2405.18540 ↗First posted 28 May 2024cs.CL cs.CR cs.LGcited by 54

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe and responsible deployment of large language models (LLMs). Developing effective protection against many modes of attack prompts requires discovering diverse attacks. Automated red-teaming typically uses reinforcement learning to fine-tune an attacker language model to generate prompts that elicit undesirable responses from a target LLM, as measured, for example, by an auxiliary toxicity classifier. We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks. As a flexible and probabilistically principled alternative, we propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts. We find that the attacks generated by our method are effective against a wide range of target LLMs, both with and without safety tuning, and transfer well between target LLMs. Finally, we demonstrate that models safety-tuned using a dataset of red-teaming prompts generated by our method are robust to attacks from other RL-based red-teaming approaches.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 55 of its 77 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

55 of 77 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Curiosity-driven Red-teaming for Large Language Models29 Feb
  2. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  4. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimization27 May
  5. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
38 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2021Ethical and social risks of harm from Language Modelscited by 957
  10. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  13. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  14. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  15. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  16. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  17. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  18. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  19. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  20. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  21. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  22. 2021GFlowNet Foundationscited by 286
  23. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  24. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  25. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  26. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  27. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  28. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  29. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  30. 2022Biological Sequence Design with GFlowNetscited by 172
  31. 2022Bayesian Structure Learning with Generative Flow Networkscited by 153
  32. 2023Amortizing intractable inference in large language modelscited by 94
  33. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  34. 2023Robust Scheduling with GFlowNetscited by 67
  35. 2022A-NeSI: A Scalable Approximate Method for Probabilistic Neurosymbolic Inferencecited by 50
  36. 2023GFlowNet-EM for learning compositional latent variable modelscited by 40
  37. 2021Analyzing Dynamic Adversarial Training Data in the Limitcited by 24
  38. 2024QGFN: Controllable Greediness with Action Valuescited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 25; 2026: 17.

54 papers.

Sort:

By area:

  1. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  2. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  3. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  4. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  5. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  6. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 29
  7. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
  8. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  9. 2024Adaptive teachers for amortized samplerscited by 23
  10. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21
  11. 2024HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Modelscited by 17
  12. 2025Learning to Sample Effective and Diverse Prompts for Text-to-Image Generationcited by 16
38 more
  1. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  2. 2025TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewardscited by 11
  3. 2024Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generationcited by 10
  4. 2025Jailbreaking on Text-to-Video Models via Scene Splitting Strategycited by 10
  5. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  6. 2025Revisiting Non-Acyclic GFlowNets in Discrete Environmentscited by 8
  7. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 8
  8. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  9. 2025Adversarial Preference Learning for Robust LLM Alignmentcited by 7
  10. 2025Adaptive Destruction Processes for Diffusion Samplerscited by 6
  11. 2024Random Policy Evaluation Uncovers Policies of Generative Flow Networkscited by 5
  12. 2025Neural Genetic Search in Discrete Spacescited by 5
  13. 2025Harmful Prompt Laundering: Jailbreaking LLMs with Abductive Styles and Symbolic Encodingcited by 5
  14. 2025Active Attacks: Red-teaming LLMs via Adaptive Environmentscited by 5
  15. 2026TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignmentcited by 5
  16. 2025CrossGuard: Safeguarding MLLMs against Joint-Modal Implicit Malicious Attackscited by 4
  17. 2024Exploring Straightforward Conversational Red-Teamingcited by 3
  18. 2025Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Expertscited by 3
  19. 2026AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attackscited by 3
  20. 2025Watch, Listen, Understand, Mislead: Tri-modal Adversarial Attacks on Short Videos for Content Appropriateness Evaluationcited by 2
  21. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  22. 2026Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teamingcited by 2
  23. 2026CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulationcited by 2
  24. 2025Can Safety Fine-Tuning Be More Principled? Lessons Learned from Cybersecuritycited by 1
  25. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  26. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  27. 2025Pattern Enhanced Multi-Turn Jailbreaking: Exploiting Structural Vulnerabilities in Large Language Modelscited by 1
  28. 2026When AI Fails, What Works? A Data-Driven Taxonomy of Real-World AI Risk Mitigation Strategiescited by 1
  29. 2026CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinementcited by 1
  30. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  31. 2025From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Modelscited by 0
  32. 2026Malicious Repurposing of Open Science Artefacts by Using Large Language Modelscited by 0
  33. 2026GFlowPO: Generative Flow Network as a Language Model Prompt Optimizercited by 0
  34. 2026Flow of Spans: Generalizing Language Models to Dynamic Span-Vocabulary via GFlowNetscited by 0
  35. 2026Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVRcited by 0
  36. 2026Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balancecited by 0
  37. 2026NodeSynth: Socially Aligned Synthetic Data for AI Evaluationcited by 0
  38. 2026FoeGlass: Simple In-Context Learning Is Enough for Red Teaming Audio Deepfake Detectorscited by 0

50 most cited of 54 All 54 (JSON)