mlsanityData to 7 Oct 2026

Testing the Limits of Jailbreaking Defenses with the Purple Problem

arXiv 2403.14725 ↗First posted 20 Mar 2024cs.CR cs.CL cs.LGcited by 8

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The rise of "jailbreak" attacks on language models has led to a flurry of defenses aimed at preventing undesirable responses. We critically examine the two stages of the defense pipeline: (i) defining what constitutes unsafe outputs, and (ii) enforcing the definition via methods such as input processing or fine-tuning. To test the efficacy of existing enforcement mechanisms, we consider a simple and well-specified definition of unsafe outputs--outputs that contain the word "purple". Surprisingly, existing fine-tuning and input defenses fail on this simple problem, casting doubt on whether enforcement algorithms can be robust for more complicated definitions. We find that real safety benchmarks similarly test enforcement for a fixed definition. We hope that future research can lead to effective/fast enforcement as well as high quality definitions used for enforcement and evaluation.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 39 of its 54 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

39 of 54 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Fast Adversarial Attacks on Language Models In One GPU Minute23 Feb
  2. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  3. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  4. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  5. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  8. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
18 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  3. 2021Ethical and social risks of harm from Language Modelscited by 957
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  5. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  6. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  7. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  8. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  9. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  10. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  11. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  12. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  13. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 124
  14. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  15. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 111
  16. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  17. 2023Toxic Bias: Perspective API Misreads German as More Toxiccited by 19
  18. 2023Critical Perspectives: A Benchmark Revealing Pitfalls in PerspectiveAPIcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 1.

8 papers.

  1. 2024Endless Jailbreaks with Bijection Learningcited by 18
  2. 2025Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?cited by 3
  3. 2024[WIP] Jailbreak Paradox: The Achilles' Heel of LLMscited by 2
  4. 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systemscited by 2
  5. 2026Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputscited by 2
  6. 2024Behavior Trees Enable Structured Programming of Language Model Agentscited by 0
  7. 2025FLAME: Flexible LLM-Assisted Moderation Enginecited by 0
  8. 2025Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integrationcited by 0