mlsanityData to 7 Oct 2026

Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF

arXiv 2405.19320 ↗First posted 29 May 2024cs.LG cs.AI stat.MLcited by 73

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has demonstrated great promise in aligning large language models (LLMs) with human preference. Depending on the availability of preference data, both online and offline RLHF are active areas of investigation. A key bottleneck is understanding how to incorporate uncertainty estimation in the reward function learned from the preference data for RLHF, regardless of how the preference data is collected. While the principles of optimism or pessimism under uncertainty are well-established in standard reinforcement learning (RL), a practically-implementable and theoretically-grounded form amenable to large language models is not yet available, as standard techniques for constructing confidence intervals become intractable under arbitrary policy parameterizations. In this paper, we introduce a unified approach to online and offline RLHF -- value-incentivized preference optimization (VPO) -- which regularizes the maximum-likelihood estimate of the reward function with the corresponding value function, modulated by a textitsign to indicate whether the optimism or pessimism is chosen. VPO also directly optimizes the policy with implicit reward modeling, and therefore shares a simpler RLHF pipeline similar to direct preference optimization. Theoretical guarantees of VPO are provided for both online and offline settings, matching the rates of their standard RL counterparts. Moreover, experiments on text summarization and dialog verify the practicality and effectiveness of VPO.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 65 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

46 of 65 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  2. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  3. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF29 Apr
  4. 2024Dataset Reset Policy Optimization for RLHF12 Apr
  5. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  6. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  7. 2023Nash Learning from Human Feedback1 Dec
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  10. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
22 more
  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  3. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  4. 2018Is Q-learning Provably Efficient?cited by 485
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  7. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  8. 2024Iterative Reasoning Preference Optimizationcited by 244
  9. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  10. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  11. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  12. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  13. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  14. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  15. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  16. 2017Boltzmann Exploration Done Rightcited by 69
  17. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  18. 2021The Power of Exploiter: Provable Multi-Agent RL in Large State Spacescited by 48
  19. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
  20. 2023Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Explorationcited by 20
  21. 2020Reward Biased Maximum Likelihood Estimation for Reinforcement Learningcited by 11
  22. 2020Reward-Biased Maximum Likelihood Estimation for Linear Stochastic Banditscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 42; 2026: 13.

73 papers.

Sort:

By area:

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  2. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  3. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  4. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  5. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  6. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  7. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  8. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  9. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  10. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 51
  11. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  12. 2025A Survey of Direct Preference Optimizationcited by 35
38 more
  1. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  2. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 24
  3. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  4. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  5. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
  6. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 18
  7. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 17
  8. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  9. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  10. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  11. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 11
  12. 2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
  13. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
  14. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuningcited by 10
  15. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  16. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  17. 2025Greedy Sampling Is Provably Efficient for RLHFcited by 10
  18. 2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignmentcited by 9
  19. 2025Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHFcited by 9
  20. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  21. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  22. 2025Mitigating Preference Hacking in Policy Optimization with Pessimismcited by 7
  23. 2025Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limitscited by 7
  24. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6
  25. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  26. 2025Repair-R1: Better Test Before Repaircited by 6
  27. 2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
  28. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  29. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
  30. 2025Design Considerations in Offline Preference-based RLcited by 4
  31. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  32. 2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
  33. 2025Offline and Online KL-Regularized RLHF under Differential Privacycited by 3
  34. 2025Improved Bounds for Private and Robust Alignmentcited by 3
  35. 2025Tokenized Bandit for LLM Decoding and Alignmentcited by 2
  36. 2025Statistical and Algorithmic Foundations of Reinforcement Learningcited by 2
  37. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
  38. 2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscited by 2

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  4. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  5. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  7. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
  8. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling