mlsanityData to 7 Oct 2026

Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study

arXiv 2404.10719 ↗First posted 16 Apr 2024cs.CLcited by 262

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is currently the most widely used method to align large language models (LLMs) with human preferences. Existing RLHF methods can be roughly categorized as either reward-based or reward-free. Novel applications such as ChatGPT and Claude leverage reward-based methods that first learn a reward model and apply actor-critic algorithms, such as Proximal Policy Optimization (PPO). However, in academic benchmarks, state-of-the-art results are often achieved via reward-free methods, such as Direct Preference Optimization (DPO). Is DPO truly superior to PPO? Why does PPO perform poorly on these benchmarks? In this paper, we first conduct both theoretical and empirical studies on the algorithmic properties of DPO and show that DPO may have fundamental limitations. Moreover, we also comprehensively examine PPO and reveal the key factors for the best performances of PPO in fine-tuning LLMs. Finally, we benchmark DPO and PPO across a collection of RLHF testbeds, ranging from dialogue to code generation. Experiment results demonstrate that PPO is able to surpass other alignment methods in all cases and achieve state-of-the-art results in challenging code competitions. Our code is publicly available at https://github.com/openpsi-project/ReaLHF.

Key facts. Cited by 262 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 62 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

48 of 62 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  2. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment24 Jul
  3. 2023Preference Ranking Optimization for Human Alignment30 Jun
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
32 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  13. 2023Instruction Tuning with GPT-4cited by 676
  14. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  15. 2024Self-Rewarding Language Modelscited by 654
  16. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  17. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  18. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  19. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  20. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  21. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  22. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  23. 2022UL2: Unifying Language Learning Paradigmscited by 292
  24. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  25. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  26. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  27. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  28. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  29. 2023Do Language Models Know When They're Hallucinating References?cited by 115
  30. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
  31. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  32. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 79; 2025: 133; 2026: 50.

262 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  4. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  5. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  6. 2024Generative Reward Modelscited by 148
  7. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  8. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  9. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  10. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  11. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  12. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
38 more
  1. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  2. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  3. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  4. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 82
  5. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  6. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  7. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  8. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  9. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  10. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  11. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  12. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  13. 2025Can Large Reasoning Models Self-Train?cited by 66
  14. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  15. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 64
  16. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  17. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  18. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  19. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
  20. 2024WPO: Enhancing RLHF with Weighted Preference Optimizationcited by 53
  21. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  22. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  23. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 49
  24. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  25. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  26. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  27. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  28. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  29. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 43
  30. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  31. 2025DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Drivingcited by 39
  32. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  33. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 36
  34. 2025A Survey of Direct Preference Optimizationcited by 35
  35. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  36. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignmentcited by 32
  37. 2024Reward-Robust RLHF in LLMscited by 32
  38. 2025Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPOcited by 31

50 most cited of 262 All 262 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Minor DPO reject penalty to increase training robustness
  2. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  3. 2024Direct Preference Optimization with an Offset
  4. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF
  5. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  6. 2024Accelerated Preference Optimization for Large Language Model Alignment
  7. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  8. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs