mlsanityData to 7 Oct 2026

Enhancing LLM Safety via Constrained Direct Preference Optimization

arXiv 2403.02475 ↗First posted 4 Mar 2024cs.LG cs.CLcited by 47

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The rapidly increasing capabilities of large language models (LLMs) raise an urgent need to align AI systems with diverse human preferences to simultaneously enhance their usefulness and safety, despite the often conflicting nature of these goals. To address this important problem, a promising approach is to enforce a safety constraint at the fine-tuning stage through a constrained Reinforcement Learning from Human Feedback (RLHF) framework. This approach, however, is computationally expensive and often unstable. In this work, we introduce Constrained DPO (C-DPO), a novel extension of the recently proposed Direct Preference Optimization (DPO) approach for fine-tuning LLMs that is both efficient and lightweight. By integrating dual gradient descent and DPO, our method identifies a nearly optimal trade-off between helpfulness and harmlessness without using reinforcement learning. Empirically, our approach provides a safety guarantee to LLMs that is missing in DPO while achieving significantly higher rewards under the same safety constraint compared to a recently proposed safe RLHF approach. Warning: This paper contains example data that may be offensive or harmful.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 47 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

32 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2021Convex optimizationcited by 2,479
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
19 more
  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  2. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  6. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  7. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  8. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  9. 2019Batch Policy Learning under Constraintscited by 242
  10. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  11. 2023Nash Learning from Human Feedbackcited by 220
  12. 2020Projection-Based Constrained Policy Optimizationcited by 195
  13. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  14. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  15. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  16. 2021A Primal-Dual Approach to Constrained Markov Decision Processescited by 42
  17. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 40
  18. 2023Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptationcited by 16
  19. 2017Safe Policy Search with Gaussian Process Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 26; 2026: 8.

47 papers.

Sort:

By area:

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  4. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  5. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  6. 2025DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learningcited by 62
  7. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 58
  8. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  9. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  10. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  11. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
  12. 2025Mesh-RFT: Enhancing Mesh Generation via Fine-grained Reinforcement Fine-Tuningcited by 21
35 more
  1. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 19
  2. 2024You Know What I'm Saying: Jailbreak Attack via Implicit Referencecited by 13
  3. 2025Learning Safety Constraints for Large Language Modelscited by 13
  4. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  5. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 10
  6. 2024Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directionscited by 10
  7. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  8. 2025Safety Alignment Can Be Not Superficial With Explicit Safety Signalscited by 9
  9. 2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimizationcited by 8
  10. 2025Alignment of large language models with constrained learningcited by 8
  11. 2025Distributional AGI Safetycited by 8
  12. 2025Auto-Connect: Connectivity-Preserving RigFormer with Direct Preference Optimizationcited by 7
  13. 2025Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safetycited by 7
  14. 2025The Geometry of Harmfulness in LLMs through Subconcept Probingcited by 7
  15. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectorscited by 6
  16. 2025Proactive defense against LLM Jailbreakcited by 6
  17. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMscited by 4
  18. 2025Reinforcement Learning from Human Feedback with High-Confidence Safety Constraintscited by 4
  19. 2025Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defensecited by 4
  20. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontiercited by 3
  21. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasingcited by 3
  22. 2025Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralizationcited by 3
  23. 2025MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Frameworkcited by 3
  24. 2025DetectiumFire: A Comprehensive Multi-modal Dataset Bridging Vision and Language for Fire Understandingcited by 2
  25. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxiescited by 2
  26. 2025Provably Convergent Primal-Dual DPO for Constrained LLM Alignmentcited by 0
  27. 2025Don't Walk the Line: Boundary Guidance for Filtered Generationcited by 0
  28. 2025Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignmentcited by 0
  29. 2026BarrierSteer: LLM Safety via Learning Barrier Steeringcited by 0
  30. 2026Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dualcited by 0
  31. 2026MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inferencecited by 0
  32. 2026BSO: Safety Alignment Is Density Ratio Matchingcited by 0
  33. 2026Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation Tracescited by 0
  34. 2026Direct Preference Density Alignment for Conversational Audio Equalizationcited by 0
  35. 2026How to train your model organismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Direct Preference Optimization
  2. 2024Minor DPO reject penalty to increase training robustness
  3. 2026Improving Safety Alignment via Balanced Direct Preference Optimization
  4. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
  5. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  6. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  7. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization
  8. 2024Active Preference Learning for Large Language Models