mlsanityData to 7 Oct 2026

Stepwise Alignment for Constrained Language Model Policy Optimization

arXiv 2404.11049 ↗First posted 17 Apr 2024cs.LG cs.AI cs.CLcited by 30

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Safety and trustworthiness are indispensable requirements for real-world applications of AI systems using large language models (LLMs). This paper formulates human value alignment as an optimization problem of the language model policy to maximize reward under a safety constraint, and then proposes an algorithm, Stepwise Alignment for Constrained Policy Optimization (SACPO). One key idea behind SACPO, supported by theory, is that the optimal policy incorporating reward and safety can be directly obtained from a reward-aligned policy. Building on this key idea, SACPO aligns LLMs step-wise with each metric while leveraging simple yet powerful alignment algorithms such as direct preference optimization (DPO). SACPO offers several advantages, including simplicity, stability, computational efficiency, and flexibility of algorithms and datasets. Under mild assumptions, our theoretical analysis provides the upper bounds on optimality and safety constraint violation. Our experimental results show that SACPO can fine-tune Alpaca-7B better than the state-of-the-art method in terms of both helpfulness and harmlessness.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 59 references that are in the map. First posted on arXiv 17 Apr 2024.

Builds on

43 of 59 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Enhancing LLM Safety via Constrained Direct Preference Optimization4 Mar
  2. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs3 Feb
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023Verbosity Bias in Preference Labeling by Large Language Models16 Oct
  5. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
25 more
  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  5. 2022PAL: Program-aided Language Modelscited by 635
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  7. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  9. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  11. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  12. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  13. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  14. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  15. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  16. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  17. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  18. 2023AI Alignment: A Comprehensive Surveycited by 256
  19. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  20. 2019Batch Policy Learning under Constraintscited by 242
  21. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  22. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  23. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 75
  24. 2021Mitigating Political Bias in Language Models Through Reinforced Calibrationcited by 73
  25. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 13.

30 papers.

Sort:

By area:

  1. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  2. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  3. 2025A Survey of Direct Preference Optimizationcited by 35
  4. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
  5. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignmentcited by 20
  6. 2025Learning Safety Constraints for Large Language Modelscited by 13
  7. 2025Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerabilitycited by 10
  8. 2025SafeSearch: Do Not Trade Safety for Utility in LLM Search Agentscited by 9
  9. 2025Alignment of large language models with constrained learningcited by 8
  10. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5
  11. 2025Reinforcement Learning from Human Feedback with High-Confidence Safety Constraintscited by 4
  12. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasingcited by 3
18 more
  1. 2025Risk-aware Direct Preference Optimization under Nested Risk Measurecited by 2
  2. 2026When Behavioral Safety Evaluation Fails: A Representation-Level Perspectivecited by 2
  3. 2026Reinforcing Real-world Service Agents: Balancing Utility and Cost in Task-oriented Dialoguecited by 1
  4. 2026What Is the Alignment Tax?cited by 1
  5. 2026Dual Optimal: Make Your LLM Peer-like with Dignitycited by 1
  6. 2026Cat-DPO: Category-Adaptive Safety Alignmentcited by 1
  7. 2025Provably Convergent Primal-Dual DPO for Constrained LLM Alignmentcited by 0
  8. 2025Don't Walk the Line: Boundary Guidance for Filtered Generationcited by 0
  9. 2025Offline Clustering of Preference Learning with Active-data Augmentationcited by 0
  10. 2025Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignmentcited by 0
  11. 2026Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dualcited by 0
  12. 2026Offline Constrained RLHF with Multiple Preference Oraclescited by 0
  13. 2026S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHFcited by 0
  14. 2026CLEAR: Continuous Latent Adapter Routing for Utility-Preserving LLM Safety Alignmentcited by 0
  15. 2026Suan: Rectifying Direct Preference Safety Alignment in Large Language Modelscited by 0
  16. 2026AUDITPLAN: Commit, Then Answer for Auditable Safety Alignmentcited by 0
  17. 2026Local Sparsity Enables Unsupervised LLM Safety Detectioncited by 0
  18. 2026How to train your model organismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
  2. 2024Towards Efficient Exact Optimization of Language Model Alignment
  3. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach
  4. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
  5. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimization
  6. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
  7. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  8. 2026BSO: Safety Alignment Is Density Ratio Matching