mlsanityData to 14 Oct 2026

Improving LLM Safety Alignment with Dual-Objective Optimization

arXiv 2503.03710 ↗First posted 5 Mar 2025cs.CL cs.CR cs.LGcited by 43

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks. Direct preference optimization (DPO), a widely deployed alignment method, exhibits limitations in both experimental and theoretical contexts as its loss function proves suboptimal for refusal learning. Through gradient-based analysis, we identify these shortcomings and propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge. This approach significantly increases LLM robustness against a wide range of jailbreak attacks, including prefilling, suffix, and multi-turn attacks across both in-distribution and out-of-distribution scenarios. Furthermore, we introduce a method to emphasize critical refusal tokens by incorporating a reward-based token-level weighting mechanism for refusal learning, which further improves the robustness against adversarial exploits. Our research also suggests that robustness to jailbreak attacks is correlated with token distribution shifts in the training process and internal representations of refusal and harmful tokens, offering valuable directions for future research in LLM safety alignment. The code is available at https://github.com/wicai24/DOOR-Alignment

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 38 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

31 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Course-Correction: Safety Alignment Using Synthetic Preferences23 Jul
  2. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal20 Jun
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  5. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  10. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
12 more
  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  2. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  3. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  5. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  6. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 158
  7. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  8. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  9. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 96
  10. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  11. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspectivecited by 69
  12. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 27.

43 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  2. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  3. 2025Model Unlearning via Sparse Autoencoder Subspace Guided Projectionscited by 18
  4. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  5. 2025Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimizationcited by 10
  6. 2025Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerabilitycited by 10
  7. 2025ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistantscited by 6
  8. 2025Read the Scene, Not the Script: Outcome-Aware Safety for LLMscited by 6
  9. 2025When Grammar Guides the Attack: Uncovering Control-Plane Vulnerabilities in LLMs with Structured Outputcited by 4
  10. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 4
  11. 2026SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignmentcited by 4
  12. 2025SEALGuard: Safeguarding the Multilingual Conversations in Southeast Asian Languages for LLM Software Systemscited by 2
31 more
  1. 2025AdaptiveGuard: Towards Adaptive Runtime Safety for LLM-Powered Softwarecited by 2
  2. 2025Dual-Space Smoothness for Robust and Balanced LLM Unlearningcited by 2
  3. 2026STaR: Sensitive Trajectory Regulation for Unlearning in Large Reasoning Modelscited by 2
  4. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  5. 2025BiasGym: A Simple and Generalizable Framework for Analyzing and Removing Biases through Injectioncited by 1
  6. 2025ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safetycited by 1
  7. 2025Why Does Weak-OOD Help? A Further Step Towards Understanding Jailbreaking VLMscited by 1
  8. 2026Improving Safety Alignment via Balanced Direct Preference Optimizationcited by 1
  9. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimizationcited by 1
  10. 2026Obliviate: Erasing Concepts from Autoregressive Image Generation Modelscited by 1
  11. 2026SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreakscited by 1
  12. 2025SafeConstellations: Mitigating Over-Refusals in LLMs Through Task-Aware Representation Steeringcited by 0
  13. 2026CuMA: Aligning LLMs with Sparse Cultural Values via Demographic-Aware Mixture of Adapterscited by 0
  14. 2026Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimizationcited by 0
  15. 2026Alignment-Aware Model Adaptation via Feedback-Guided Optimizationcited by 0
  16. 2026Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routingcited by 0
  17. 2026Structured Semantic Cloaking for Jailbreak Attacks on Large Language Modelscited by 0
  18. 2026Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidancecited by 0
  19. 2026Curriculum Learning for Safety Alignmentcited by 0
  20. 2026DDOR: Delta Debugging for Explainable Overrefusal Testing and Repaircited by 0
  21. 2026Rapid Poison: Practical Poisoning Attacks Against the Rapid Response Frameworkcited by 0
  22. 2026Adaptive and Explicit safe: Triggering Latent Safety Awareness in Large Reasoning Modelscited by 0
  23. 2026Addressing Over-Refusal in LLMs with Competing Rewardscited by 0
  24. 2026Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Modelscited by 0
  25. 2026CS-Guard: Benchmarking LLM Guardrails for Code Generation Securitycited by 0
  26. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0
  27. 2026MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Modelscited by 0
  28. 2026TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safetycited by 0
  29. 2026SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Modelscited by 0
  30. 2026How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defensecited by 0
  31. 2026When Protection Overrides Consequences: Understanding Protection Mode in Large Language Model Decision-Makingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMs
  2. 2026Fail-Closed Alignment for Large Language Models
  3. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignment
  4. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMs
  5. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  6. 2024Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment
  7. 2026Understanding the Effects of Safety Unalignment on Large Language Models
  8. 2025Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Direction