mlsanityData to 7 Oct 2026

Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences

arXiv 2403.01857 ↗First posted 4 Mar 2024cs.LGcited by 27

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In this paper, we take a step towards a deeper understanding of learning from human preferences by systematically comparing the paradigm of reinforcement learning from human feedback (RLHF) with the recently proposed paradigm of direct preference optimization (DPO). We focus our attention on the class of loglinear policy parametrization and linear reward functions. In order to compare the two paradigms, we first derive minimax statistical bounds on the suboptimality gap induced by both RLHF and DPO, assuming access to an oracle that exactly solves the optimization problems. We provide a detailed discussion on the relative comparison between the two paradigms, simultaneously taking into account the sample size, policy and reward class dimensions, and the regularization temperature. Moreover, we extend our analysis to the approximate optimization setting and derive exponentially decaying convergence rates for both RLHF and DPO. Next, we analyze the setting where the ground-truth reward is not realizable and find that, while RLHF incurs a constant additional error, DPO retains its asymptotically decaying gap by just tuning the temperature accordingly. Finally, we extend our comparison to the Markov decision process setting, where we generalize our results with exact optimization. To the best of our knowledge, we are the first to provide such a comparative analysis for RLHF and DPO.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 54 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

47 of 54 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023Direct Preference-based Policy Optimization without Reward Modeling30 Jan
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  6. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  10. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  7. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  10. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  11. 2022Teaching language models to support answers with verified quotescited by 290
  12. 2021Recursively Summarizing Books with Human Feedbackcited by 287
24 more
  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  4. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  5. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  6. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  8. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  9. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  10. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  11. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  12. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  13. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  14. 2020Improved Optimistic Algorithms for Logistic Banditscited by 117
  15. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  16. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  17. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  18. 2014Reducing Dueling Bandits to Cardinal Banditscited by 62
  19. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 59
  20. 2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 51
  21. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  22. 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
  23. 2016A Relative Exponential Weighing Algorithm for Adversarial Utility-based Dueling Banditscited by 31
  24. 2018Active Ranking with Subset-wise Preferencescited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 12; 2026: 6.

27 papers.

Sort:

By area:

  1. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  2. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  3. 2025A Survey of Direct Preference Optimizationcited by 35
  4. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  5. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  6. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  7. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  8. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  9. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  10. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  11. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  12. 2024Direct Alignment of Language Models via Quality-Aware Self-Refinementcited by 6
15 more
  1. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6
  2. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
  3. 2025Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraintscited by 4
  4. 2025VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RLcited by 3
  5. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
  6. 2024Multi-Player Approaches for Dueling Banditscited by 2
  7. 2025Policy Teaching via Data Poisoning in Learning from Human Preferencescited by 2
  8. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?cited by 2
  9. 2026On the Convergence of Self-Improving Online LLM Alignmentcited by 1
  10. 2025Online Distributionally Robust LLM Alignment via Regression to Relative Rewardcited by 0
  11. 2026Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedbackcited by 0
  12. 2026Distributionally Robust Listwise Preference Optimizationcited by 0
  13. 2026Meta-Learning Preferences for Multilingual LLM Alignmentcited by 0
  14. 2026Towards a theory of inference-time alignment with unknown rewardscited by 0
  15. 2026RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
  3. 2024New Desiderata for Direct Preference Optimization
  4. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
  5. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  6. 2025Reinforcement Learning from Human Feedback
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2025Active Learning for Direct Preference Optimization