mlsanityData to 14 Oct 2026

Alignment of large language models with constrained learning

arXiv 2505.19387 ↗First posted 26 May 2025cs.LG cs.SY eess.SYcited by 8

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We study the problem of computing an optimal large language model (LLM) policy for the constrained alignment problem, where the goal is to maximize a primary reward objective while satisfying constraints on secondary utilities. Despite the popularity of Lagrangian-based LLM policy search in constrained alignment, iterative primal-dual methods often fail to converge, and non-iterative dual-based methods do not achieve optimality in the LLM parameter space. To address these challenges, we employ Lagrangian duality to develop an iterative dual-based alignment method that alternates between updating the LLM policy via Lagrangian maximization and updating the dual variable via dual descent. In theory, we characterize the primal-dual gap between the primal value in the distribution space and the dual value in the LLM parameter space. We further quantify the optimality gap of the learned LLM policies at near-optimal dual variables with respect to both the objective and the constraint functions. These results prove that dual-based alignment methods can find an optimal constrained LLM policy, up to an LLM parametrization gap. We demonstrate the effectiveness and merits of our approach through extensive experiments conducted on the PKU-SafeRLHF and Anthropic HH-RLHF datasets.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 47 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

35 of 47 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Stepwise Alignment for Constrained Language Model Policy Optimization17 Apr
  4. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models25 Mar
  5. 2024Enhancing LLM Safety via Constrained Direct Preference Optimization4 Mar
  6. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  8. 2023Alignment for Honesty12 Dec
  9. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis16 Oct
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2021Convex optimizationcited by 2,515
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  8. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  9. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  10. 2022PAL: Program-aided Language Modelscited by 652
  11. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  12. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
13 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  2. 2023Gender bias and stereotypes in Large Language Modelscited by 187
  3. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  4. 2018Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goalscited by 103
  5. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  6. 2020Probably Approximately Correct Constrained Learningcited by 47
  7. 2021Constrained Learning with Non-Convex Lossescited by 42
  8. 2024Moral Alignment for LLM Agentscited by 39
  9. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
  10. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 22
  11. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 20
  12. 2024Near-Optimal Solutions of Constrained Learning Problemscited by 9
  13. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026LACONIC: Length-Aware Constrained Reinforcement Learning for LLMcited by 4
  2. 2026HIPO: Instruction Hierarchy via Constrained Reinforcement Learningcited by 3
  3. 2026RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimizationcited by 2
  4. 2025Dual Optimistic Ascent (PI Control) is the Augmented Lagrangian Method in Disguisecited by 1
  5. 2026Everywhere Learning: Artificial Intelligence with Pointwise Constraintscited by 1
  6. 2026Risk-Conditioned Fine-Tuning of Large Language Modelscited by 1
  7. 2025Provably Convergent Primal-Dual DPO for Constrained LLM Alignmentcited by 0
  8. 2026Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dualcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024L3Ms -- Lagrange Large Language Models
  2. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  3. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
  4. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  5. 2026Theoretical Limits of Language Model Alignment
  6. 2024Efficient Alignment of Large Language Models via Data Sampling
  7. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  8. 2025LLM Alignment as Retriever Optimization: An Information Retrieval Perspective