mlsanityData to 14 Oct 2026

Policy Gradient Primal-Dual Method for Safe Reinforcement Learning from Human Feedback

arXiv 2604.19024 ↗First posted 21 Apr 2026cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Safe Reinforcement Learning from Human Feedback (Safe RLHF) has recently achieved empirical success in developing helpful and harmless large language models by decoupling human preferences regarding helpfulness and harmlessness. Existing approaches typically rely on fitting fixed horizon reward models from human feedback and have only been validated empirically. In this paper, we formulate safe RLHF as an infinite horizon discounted Con- strained Markov Decision Process (CMDP), since humans may interact with the model over a continuing sequence of interactions rather than within a single finite episode. We propose two Safe RLHF algorithms that do not require reward model fitting and, in contrast to prior work assuming fixed-length trajectories, support flexible trajectory lengths for training. Both algo- rithms are based on the primal-dual method and achieve global convergence guarantees with polynomial rates in terms of policy gradient iterations, trajectory sample lengths, and human preference queries. To the best of our knowledge, this is the first work to study infinite horizon discounted CMDP under human feedback and establish global, non-asymptotic convergence.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 38 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

26 of 38 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference25 Sep
  2. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis11 Jun
  3. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization15 Feb
  4. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  6. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 17

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  7. 2018Reward Constrained Policy Optimizationcited by 417
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  9. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  10. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 298
  11. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  12. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 111
5 more
  1. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  2. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  3. 2020A Sample-Efficient Algorithm for Episodic Finite-Horizon MDP with Constraintscited by 34
  4. 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
  5. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settingscited by 1
  2. 2026Understanding the Impact of AI Code Assistants on Security API Usage: An Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning from Human Feedback with High-Confidence Safety Constraints
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  4. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  5. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  7. 2023SuperHF: Supervised Iterative Learning from Human Feedback
  8. 2023A Survey of Reinforcement Learning from Human Feedback