mlsanityData to 7 Oct 2026

Purple-teaming LLMs with Adversarial Defender Training

arXiv 2407.01850 ↗First posted 1 Jul 2024cs.CLcited by 4

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Existing efforts in safeguarding LLMs are limited in actively exposing the vulnerabilities of the target LLM and readily adapting to newly emerging safety risks. To address this, we present Purple-teaming LLMs with Adversarial Defender training (PAD), a pipeline designed to safeguard LLMs by novelly incorporating the red-teaming (attack) and blue-teaming (safety training) techniques. In PAD, we automatically collect conversational data that cover the vulnerabilities of an LLM around specific safety risks in a self-play manner, where the attacker aims to elicit unsafe responses and the defender generates safe responses to these attacks. We then update both modules in a generative adversarial network style by training the attacker to elicit more unsafe responses and updating the defender to identify them and explain the unsafe reason. Experimental results demonstrate that PAD significantly outperforms existing baselines in both finding effective attacks and establishing a robust safe guardrail. Furthermore, our findings indicate that PAD excels in striking a balance between safety and overall model quality. We also reveal key challenges in safeguarding LLMs, including defending multi-turn attacks and the need for more delicate strategies to identify specific risks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 36 of its 56 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

36 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  3. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 33

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
21 more
  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2018AI safety via debatecited by 384
  5. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  6. 2022Data Contamination: From Memorization to Exploitationcited by 234
  7. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  8. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  9. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  10. 2022COLD: A Benchmark for Chinese Offensive Language Detectioncited by 74
  11. 2022Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identitycited by 71
  12. 2021Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contextscited by 56
  13. 2023Self-Guard: Empower the LLM to Safeguard Itselfcited by 52
  14. 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectorscited by 48
  15. 2021SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failurescited by 37
  16. 2022ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situationscited by 36
  17. 2023Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?cited by 36
  18. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34
  19. 2022Mitigating Covertly Unsafe Text within Natural Language Systemscited by 13
  20. 2023CLEVA: Chinese Language Models EVAluation Platformcited by 13
  21. 2024TroubleLLM: Align to Red Team Expertcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 3.

4 papers.

  1. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  2. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 13
  3. 2026Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teamingcited by 1
  4. 2026Persona-Conditioned Adversarial Prompting: Multi-Identity Red-Teaming for Adversarial Discovery and Mitigationcited by 0