Data to 7 Oct 2026

Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

arXiv 2406.09279 ↗First posted 13 Jun 2024cs.CLcited by 107

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories. We publicly release the code used for training (https://github.com/hamishivi/EasyLM) and evaluating (https://github.com/allenai/open-instruct) our models, along with the models and datasets themselves (https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618).

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 55 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

36 of 55 references.

Closest ancestors · 2

Same direction, newest first

2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
2017Proximal Policy Optimization Algorithms20 Jul

Other references · 34

2022Training language models to follow instructions with human feedbackcited by 15,712
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2021Evaluating Large Language Models Trained on Codecited by 8,053
2020Measuring Massive Multitask Language Understandingcited by 7,768
2017Deep reinforcement learning from human preferencescited by 3,846
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2021Program Synthesis with Large Language Modelscited by 3,509
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
2019Fine-Tuning Language Models from Human Preferencescited by 2,124

22 more

2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
2024Understanding the performance gap between online and offline alignment algorithmscited by 110
2022Rainier: Reinforced Knowledge Introspector for Commonsense Question Answeringcited by 43
2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Taskscited by 35
2023Crystal: Introspective Reasoners Reinforced with Self-Feedbackcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 63; 2026: 23.

107 papers.

Sort:

By area:

2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
2025RM-R1: Reward Modeling as Reasoningcited by 149
2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50

38 more

2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
2025A Survey of Direct Preference Optimizationcited by 35
2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
2025TESS 2: A Large-Scale Generalist Diffusion Language Modelcited by 30
2024Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMscited by 28
2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
2025Exploiting Tree Structure for Credit Assignment in RL Training of LLMscited by 27
2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
2024Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignmentcited by 24
2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Methodcited by 22
2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
2025Differentially Private Steering for Large Language Model Alignmentcited by 15
2026Subliminal Effects in Your Data: A General Mechanism via Log-Linearitycited by 15
2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimizationcited by 14
2025Rethinking DPO: The Role of Rejected Responses in Preference Misalignmentcited by 14
2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
2025MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Trainingcited by 12
2025WorldPM: Scaling Human Preference Modelingcited by 12
2025APT: Improving Specialist LLM Performance with Weakness Case Acquisition and Iterative Preference Trainingcited by 12
2025InSPO: Unlocking Intrinsic Self-Reflection for LLM Preference Optimizationcited by 11
2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignmentcited by 10
2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimizationcited by 10
2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
2025What Matters in Data for DPO?cited by 10
2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modelingcited by 10
2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimizationcited by 9
2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignmentcited by 9
2025Aligning Large Language Models with Implicit Preferences from User-Generated Contentcited by 9
2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 9
2024A Systematic Examination of Preference Learning through the Lens of Instruction-Followingcited by 8
2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Datasetcited by 8

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way