mlsanityData to 7 Oct 2026

Offline Constrained RLHF with Multiple Preference Oracles

arXiv 2604.00200 ↗First posted 31 Mar 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 53 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

36 of 53 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Axioms for AI Alignment from Human Feedback23 May
  2. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization15 Feb
  3. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences14 Feb
  4. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  6. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  7. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback3 Aug
  8. 2023Is RLHF More Difficult than Standard RL?25 Jun
  9. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  10. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  11. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  5. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  6. 2024Reinforcement learningcited by 236
  7. 2020Exploration-Exploitation in Constrained MDPscited by 146
  8. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  9. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  10. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  11. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  12. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
12 more
  1. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  2. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  3. 2020Constrained Upper Confidence Reinforcement Learningcited by 46
  4. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 33
  5. 2016Learning Preferences for Manipulation Tasks from Online Coactive Feedbackcited by 31
  6. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  7. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
  8. 2023A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learningcited by 9
  9. 2023Fairness in Preference-based Reinforcement Learningcited by 9
  10. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  11. 2025Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Functioncited by 4
  12. 2025Offline Constrained Reinforcement Learning under Partial Data Coveragecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
  2. 2024On the Exponential Convergence for Offline RLHF with Pairwise Comparisons
  3. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  4. 2025Online Optimization for Offline Safe Reinforcement Learning
  5. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithms
  6. 2024A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs
  7. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
  8. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning