mlsanityData to 7 Oct 2026

Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training

arXiv 2407.09121 ↗First posted 12 Jul 2024cs.CL cs.AIcited by 67

Not on the map. Why?

Cited by 38 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs) by identifying and tackling a refusal position bias within safety tuning data, which compromises the models' ability to appropriately refuse generating unsafe content. We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position, significantly enhancing their safety capabilities. DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model families across six attack scenarios, demonstrates that our method not only improves model safety without compromising performance but also surpasses baseline methods in defending against attacks.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 54 of its 82 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

54 of 82 references.

References · 54

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
38 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  9. 2024Self-Rewarding Language Modelscited by 654
  10. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  11. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  12. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  13. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  14. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  15. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  16. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  17. 2023Detecting Language Model Attacks with Perplexitycited by 411
  18. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  19. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  20. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  21. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  22. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  23. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  24. 2023Pretraining Language Models with Human Preferencescited by 245
  25. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  26. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  27. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  28. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  29. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  30. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  31. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  32. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  33. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 79
  34. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
  35. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
  36. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  37. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 64
  38. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapescited by 63

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 34; 2026: 24.

67 papers.

Sort:

By area:

  1. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  2. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  3. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 37
  4. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  5. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 23
  6. 2024Locking Down the Finetuned LLMs Safetycited by 22
  7. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 22
  8. 2025X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usabilitycited by 20
  9. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20
  10. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  11. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  12. 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 20
38 more
  1. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  2. 2024Endless Jailbreaks with Bijection Learningcited by 18
  3. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  4. 2024HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Routercited by 16
  5. 2026Intelligent AI Delegationcited by 16
  6. 2025Safety Reasoning with Guidelinescited by 14
  7. 2024Course-Correction: Safety Alignment Using Synthetic Preferencescited by 13
  8. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  9. 2025Lifelong Safety Alignment for Language Modelscited by 13
  10. 2025Too Good to be Bad: On the Failure of LLMs to Role-Play Villainscited by 10
  11. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signalscited by 9
  12. 2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimizationcited by 8
  13. 2025Jailbreaking to Jailbreakcited by 8
  14. 2025Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Modelscited by 8
  15. 2026TraceRouter: Robust Safety for Large Foundation Models via Path-Level Interventioncited by 8
  16. 2024Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regressioncited by 6
  17. 2025Read the Scene, Not the Script: Outcome-Aware Safety for LLMscited by 6
  18. 2026What Matters For Safety Alignment?cited by 6
  19. 2025HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humorcited by 5
  20. 2025Trustworthy AI: Safety, Bias, and Privacy -- A Surveycited by 5
  21. 2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalitiescited by 5
  22. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attackscited by 4
  23. 2026Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counselingcited by 4
  24. 2024Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMscited by 3
  25. 2026Robust Policy Optimization to Prevent Catastrophic Forgettingcited by 3
  26. 2026AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attackscited by 3
  27. 2025RIDE: Enhancing Large Language Model Alignment through Restyled In-Context Learning Demonstration Exemplarscited by 2
  28. 2025Practical Principles for AI Cost and Compute Accountingcited by 2
  29. 2025Phonetic Perturbations Reveal Tokenizer-Rooted Safety Gaps in LLMscited by 2
  30. 2025A Red Teaming Roadmap Towards System-Level Safetycited by 2
  31. 2025ConceptGuard: Neuro-Symbolic Safety Guardrails via Sparse Interpretable Jailbreak Conceptscited by 2
  32. 2025How Catastrophic is Your LLM? Certifying Risk in Conversationcited by 2
  33. 2026Position: Capability Control Should be a Separate Goal From Alignmentcited by 2
  34. 2026NeST: Neuron Selective Tuning for LLM Safetycited by 2
  35. 2025Safe Vision-Language Models via Unsafe Weights Manipulationcited by 1
  36. 2025BiasGym: A Simple and Generalizable Framework for Analyzing and Removing Biases through Injectioncited by 1
  37. 2025Reflection-Driven Control for Trustworthy Code Agentscited by 1
  38. 2026Improving Safety Alignment via Balanced Direct Preference Optimizationcited by 1

50 most cited of 67 All 67 (JSON)