mlsanityData to 14 Oct 2026

One-Shot Safety Alignment for Large Language Models via Optimal Dualization

arXiv 2405.19544 ↗First posted 29 May 2024cs.AI cs.CL cs.LGcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The growing safety concerns surrounding large language models raise an urgent need to align them with diverse human preferences to simultaneously enhance their helpfulness and safety. A promising approach is to enforce safety constraints through Reinforcement Learning from Human Feedback (RLHF). For such constrained RLHF, typical Lagrangian-based primal-dual policy optimization methods are computationally expensive and often unstable. This paper presents a perspective of dualization that reduces constrained alignment to an equivalent unconstrained alignment problem. We do so by pre-optimizing a smooth and convex dual function that has a closed form. This shortcut eliminates the need for cumbersome primal-dual policy iterations, greatly reducing the computational burden and improving training stability. Our strategy leads to two practical algorithms in model-based and preference-based settings (MoCAN and PeCAN, respectively). A broad range of experiments demonstrate the effectiveness and merits of our algorithms.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 48 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

32 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  12. 2022PAL: Program-aided Language Modelscited by 652
17 more
  1. 2014Convex Optimization: Algorithms and Complexitycited by 575
  2. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  3. 2019Challenges of Real-World Reinforcement Learningcited by 291
  4. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  5. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  6. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  8. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  9. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  10. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  11. 2018Negative Momentum for Improved Game Dynamicscited by 60
  12. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 49
  13. 2024Uncertainty in Language Models: Assessment through Rank-Calibrationcited by 41
  14. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  15. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  16. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 28
  17. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 7.

24 papers.

Sort:

By area:

  1. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 49
  2. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  3. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 13
  4. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  5. 2025Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Modelscited by 10
  6. 2025Bounded Rationality for LLMs: Satisficing Alignment at Inference-Timecited by 9
  7. 2025SafeSearch: Do Not Trade Safety for Utility in LLM Search Agentscited by 9
  8. 2025Alignment of large language models with constrained learningcited by 8
  9. 2024Jailbreaking LLM-Controlled Robotscited by 6
  10. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5
  11. 2025Reinforcement Learning from Human Feedback with High-Confidence Safety Constraintscited by 4
  12. 2026SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignmentcited by 4
12 more
  1. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasingcited by 3
  2. 2025Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Modelscited by 2
  3. 2026Cat-DPO: Category-Adaptive Safety Alignmentcited by 1
  4. 2025Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learningcited by 0
  5. 2025MEUV: Achieving Fine-Grained Capability Activation in Large Language Models via Mutually Exclusive Unlock Vectorscited by 0
  6. 2025Provably Convergent Primal-Dual DPO for Constrained LLM Alignmentcited by 0
  7. 2025Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignmentcited by 0
  8. 2026Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dualcited by 0
  9. 2026Guardrails in Logit Space: Safety Token Regularization for LLM Alignmentcited by 0
  10. 2026BSO: Safety Alignment Is Density Ratio Matchingcited by 0
  11. 2026MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Modelscited by 0
  12. 2026Safety Cost of Steering Vectors Is Separable and Reduciblecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Principled Loss Function for Direct Language Model Alignment
  2. 2025Doubly Robust Alignment for Large Language Models
  3. 2026Improving Safety Alignment via Balanced Direct Preference Optimization
  4. 2025Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling
  5. 2024Towards Efficient Exact Optimization of Language Model Alignment
  6. 2026Multilingual Safety Alignment Via Sparse Weight Editing
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback