mlsanityData to 14 Oct 2026

Policy Optimization in RLHF: The Impact of Out-of-preference Data

arXiv 2312.10584 ↗First posted 17 Dec 2023cs.LGcited by 35

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Aligning intelligent agents with human preferences and values is important. This paper examines two popular alignment methods: Direct Preference Optimization (DPO) and Reward-Model-Based Policy Optimization (RMB-PO). A variant of RMB-PO, referred to as RMB-PO+ is also considered. These methods, either explicitly or implicitly, learn a reward model from preference data and differ in the data used for policy optimization to unlock the generalization ability of the reward model. In particular, compared with DPO, RMB-PO additionally uses policy-generated data, and RMB-PO+ further leverages new, preference-free data. We examine the impact of such out-of-preference data. Our study, conducted through controlled and synthetic experiments, demonstrates that DPO performs poorly, whereas RMB-PO+ performs the best. In particular, even when providing the policy model with a good feature representation, we find that policy optimization with adequate out-of-preference data significantly improves performance by harnessing the reward model's generalization capabilities.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 37 references that are in the map. First posted on arXiv 17 Dec 2023.

Builds on

23 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  4. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  7. 2016Generative Adversarial Imitation Learningcited by 1,597
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  11. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  12. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
6 more
  1. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  2. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  3. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
  4. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  5. 2022Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysiscited by 13
  6. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 18; 2025: 12; 2026: 3.

35 papers.

Sort:

By area:

  1. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  2. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  3. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  4. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  5. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 56
  6. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 51
  7. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  8. 2025A Survey of Direct Preference Optimizationcited by 36
  9. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  10. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  11. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  12. 2024Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignmentcited by 25
23 more
  1. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  2. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewardscited by 23
  3. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumcited by 23
  4. 2024Self-Improving Robust Preference Optimizationcited by 19
  5. 2024Toward Optimal LLM Alignments Using Two-Player Gamescited by 13
  6. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 13
  7. 2024D2PO: Discriminator-Guided DPO with Response Evaluation Modelscited by 9
  8. 2024ITERTL: An Iterative Framework for Fine-tuning LLMs for RTL Code Generationcited by 9
  9. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithmscited by 8
  10. 2025Towards Self-Improvement of Diffusion Models via Group Preference Optimizationcited by 7
  11. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 7
  12. 2024Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimationcited by 6
  13. 2025Explicit Preference Optimization: No Need for an Implicit Reward Modelcited by 6
  14. 2024New Desiderata for Direct Preference Optimizationcited by 5
  15. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5
  16. 2025Controlling Large Language Model with Latent Actionscited by 5
  17. 2025IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignmentcited by 3
  18. 2025Improved Algorithms for Differentially Private Language Model Alignmentcited by 2
  19. 2026FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignmentcited by 2
  20. 2025SGPO: Self-Generated Preference Optimization based on Self-Improvercited by 1
  21. 2025AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimizationcited by 0
  22. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  23. 2026Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  2. 2025Value-Free Policy Optimization via Reward Partitioning
  3. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  4. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  5. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  7. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
  8. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences