mlsanityData to 14 Oct 2026

Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO

arXiv 2505.19770 ↗First posted 26 May 2025cs.LG cs.CLcited by 10

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We present a fine-grained theoretical analysis of the performance gap between two-stage reinforcement learning from human feedback~(RLHF) and direct preference optimization~(DPO). Our study decomposes this gap into two sources: the explicit representation gap under exact optimization and the implicit representation gap under finite samples. In the exact optimization setting, we characterize how the relative capacities of the reward and policy model classes influence the final policy qualities. We show that RLHF, DPO, or online DPO can outperform one another depending on type of model mis-specifications. Notably, online DPO can outperform both RLHF and standard DPO when the reward and policy model classes are isomorphic and both mis-specified. In the approximate optimization setting, we provide a concrete construction where the ground-truth reward is sparse and show that RLHF requires significantly fewer samples than DPO to recover an effective reward model, highlighting a statistical advantage of two-stage learning. Together, these results provide a comprehensive understanding of the performance gap between RLHF and DPO under various settings, and offer practical insights into when each method is preferred.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 70 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

54 of 70 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspective19 Mar
  2. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback11 Mar
  3. 2025Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Exploration10 Mar
  4. 2025Distributionally Robust Reinforcement Learning with Human Feedback1 Mar
  5. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective26 Feb
  6. 2025Provably Efficient Online RLHF with One-Pass Reward Modeling11 Feb
  7. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling6 Feb
  8. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Exploration2 Feb
  9. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  10. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  11. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  12. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
11 more
  1. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences4 Mar
  2. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  4. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  10. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  11. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  7. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  8. 2025Process Reinforcement through Implicit Rewardscited by 504
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  10. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  11. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  12. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
19 more
  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  2. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  3. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  4. 2024Free Process Rewards without Process Labelscited by 165
  5. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
  6. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  7. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 94
  8. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  9. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspectivecited by 69
  10. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  11. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 61
  12. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  13. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  14. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  15. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48
  16. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 27
  17. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Levelcited by 22
  18. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Modelscited by 20
  19. 2025Leveraging Sparsity for Sample-Efficient Preference Learning: A Theoretical Perspectivecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  2. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  4. 2025Why DPO is a Misspecified Estimator and How to Fix Itcited by 3
  5. 2026Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignmentcited by 1
  6. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedbackcited by 0
  7. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  8. 2026Disentangling Deception and Hallucination Failures in LLMscited by 0
  9. 2026DDO-RM: Distribution-Level Policy Improvement after Reward Learningcited by 0
  10. 2026Improving LLM Collaboration via Multi-Agent Preference Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
  2. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
  3. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  4. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
  5. 2024New Desiderata for Direct Preference Optimization
  6. 2024Filtered Direct Preference Optimization
  7. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  8. 2025Reinforcement Learning from Human Feedback