mlsanityData to 7 Oct 2026

Don't Say No: Jailbreaking LLM by Suppressing Refusal

arXiv 2404.16369 ↗First posted 25 Apr 2024cs.CLcited by 53

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Ensuring the safety alignment of Large Language Models (LLMs) is critical for generating responses consistent with human values. However, LLMs remain vulnerable to jailbreaking attacks, where carefully crafted prompts manipulate them into producing toxic content. One category of such attacks reformulates the task as an optimization problem, aiming to elicit affirmative responses from the LLM. However, these methods heavily rely on predefined objectionable behaviors, limiting their effectiveness and adaptability to diverse harmful queries. In this study, we first identify why the vanilla target loss is suboptimal and then propose enhancements to the loss objective. We introduce DSN (Don't Say No) attack, which combines a cosine decay schedule method with refusal suppression to achieve higher success rates. Extensive experiments demonstrate that DSN outperforms baseline attacks and achieves state-of-the-art attack success rates (ASR). DSN also shows strong universality and transferability to unseen datasets and black-box models.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 69 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

39 of 69 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!19 Feb
  2. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  3. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  7. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
20 more
  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  2. 2016Adversarial Machine Learning at Scalecited by 934
  3. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  4. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  5. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  6. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  9. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  10. 2019The many Shapley values for model explanationcited by 341
  11. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  12. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  13. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  14. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  15. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  16. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 128
  17. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
  18. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  19. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
  20. 2024Rethinking How to Evaluate Language Model Jailbreakcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 21; 2026: 17.

53 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  3. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 58
  4. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  5. 2025Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysiscited by 33
  6. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  7. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  8. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  9. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
  10. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 16
  11. 2024Adversarial Attacks on Large Language Models Using Regularized Relaxationcited by 12
  12. 2025Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Trainingcited by 12
38 more
  1. 2024Jailbreaking as a Reward Misspecification Problemcited by 11
  2. 2024HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoningcited by 11
  3. 2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimizationcited by 8
  4. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  5. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  6. 2025LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arenacited by 6
  7. 2026Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMscited by 6
  8. 2025TurboFuzzLLM: Turbocharging Mutation-based Fuzzing for Effectively Jailbreaking Large Language Models in Practicecited by 5
  9. 2025CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMscited by 5
  10. 2025Large Language Model Sourcing: A Surveycited by 5
  11. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  12. 2024Logicbreaks: A Framework for Understanding Subversion of Rule-based Inferencecited by 3
  13. 2024LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Secondscited by 3
  14. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMscited by 3
  15. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failurescited by 3
  16. 2025Breaking Audio Large Language Models by Attacking Only the Encoder: A Universal Targeted Latent-Space Audio Attackcited by 3
  17. 2026AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attackscited by 3
  18. 2025NonTextual Target Attackcited by 2
  19. 2026Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Modelscited by 2
  20. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  21. 2024Inner-Probe: Discovering Copyright-related Data Generation in LLM Architecturecited by 1
  22. 2025SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Modelscited by 1
  23. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  24. 2025One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMscited by 1
  25. 2026The Salami Slicing Threat: Exploiting Cumulative Risks in LLM Systemscited by 1
  26. 2026CASCADE Against Jailbreaks: Combination Across Stages with Controlled Attack-Defense Evaluationcited by 1
  27. 2025From job titles to jawlines: Using context voids to study generative AI systemscited by 0
  28. 2025From Rogue to Safe AI: The Role of Explicit Refusals in Aligning LLMs with International Humanitarian Lawcited by 0
  29. 2025Leveraging the Potential of Prompt Engineering for Hate Speech Detection in Low-Resource Languagescited by 0
  30. 2025ForgeDAN: An Evolutionary Framework for Jailbreaking Aligned Large Language Modelscited by 0
  31. 2026Sensitivity-Aware Retrieval-Augmented Intent Clarificationcited by 0
  32. 2026Structured Visual Narratives Undermine Safety Alignment in Multimodal Large Language Modelscited by 0
  33. 2026Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mindcited by 0
  34. 2026GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimizationcited by 0
  35. 2026EVA: Editing for Versatile Alignment against Jailbreakscited by 0
  36. 2026Adaptive Probe-based Steering for Robust LLM Jailbreakingcited by 0
  37. 2026DoubtProbe: Black-Box Jailbreak Defense via Structural Verification and Semantic Auditingcited by 0
  38. 2026Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agentscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models
  2. 2026Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-off
  3. 2025Improving LLM Safety Alignment with Dual-Objective Optimization
  4. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment
  5. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
  6. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  7. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  8. 2025Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs