mlsanityData to 7 Oct 2026

Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dual

arXiv 2602.22146 ↗First posted 25 Feb 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) plays a significant role in aligning Large Language Models (LLMs) with human preferences. While RLHF with expected reward constraints can be formulated as a primal-dual optimization problem, standard primal-dual methods only guarantee convergence with a distributional policy where the saddle-point problem is in convex-concave form. Moreover, standard primal-dual methods may exhibit instability or divergence in the last iterate under policy parameterization in practical applications. In this work, we propose a universal primal-dual framework for safe RLHF that unifies a broad class of existing alignment algorithms, including safe-RLHF, one-shot, and multi-shot based methods. Building on this framework, we introduce an optimistic primal-dual (OPD) algorithm that incorporates predictive updates for both primal and dual variables to stabilize saddle-point dynamics. We establish last-iterate convergence guarantees for the proposed method, covering both exact policy optimization in the distributional space and convergence to a neighborhood of the optimal solution whose gap is related to approximation error and bias under parameterized policies. Our analysis reveals that optimism plays a crucial role in mitigating oscillations inherent to constrained alignment objectives, thereby closing a key theoretical gap between constrained RL and practical RLHF.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 31 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

25 of 31 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Alignment of large language models with constrained learning26 May
  2. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models16 May
  3. 2024L3Ms -- Lagrange Large Language Models28 Oct
  4. 2024Stepwise Alignment for Constrained Language Model Policy Optimization17 Apr
  5. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  6. 2024Enhancing LLM Safety via Constrained Direct Preference Optimization4 Mar
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 15

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  6. 2024A Roadmap to Pluralistic Alignmentcited by 175
  7. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  8. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  9. 2018The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimizationcited by 98
  10. 2023Unveiling the Implicit Toxicity in Large Language Modelscited by 51
  11. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  12. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26
3 more
  1. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
  2. 2024Trustworthy AI: Securing Sensitive Data in Large Language Modelscited by 7
  3. 2022Convergence rate analysis of the gradient descent-ascent method for convex-concave saddle-point problemscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Online Distributionally Robust LLM Alignment via Regression to Relative Reward
  2. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  3. 2024Accelerated Preference Optimization for Large Language Model Alignment
  4. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
  5. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
  6. 2025Provably Convergent Primal-Dual DPO for Constrained LLM Alignment
  7. 2025Doubly Robust Alignment for Large Language Models
  8. 2026Unifying Stable Optimization and Reference Regularization in RLHF