mlsanityData to 14 Oct 2026

Understanding the Impact of Sampling Quality in Direct Preference Optimization

arXiv 2506.04272 ↗First posted 3 Jun 2025cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We study how data of higher quality can be leveraged to improve performance in Direct Preference Optimization (DPO), aiming to understand its impact on DPO training dynamics. Our analyses show that both the solution space and the convergence behavior of DPO depend on the support and quality of the data-generating distribution. We first analyze how data and reference policy influence policy updates during gradient descent, and how a practical phenomenon known as likelihood displacement can interfere with the desired dynamics. We then design a simplified yet well-structured alignment model as a proxy that preserves most of the beneficial properties of RLHF while avoiding likelihood displacement. Based on this model, we develop quantitative results showing how more frequent high-quality responses amplify the gradient signal and improve the optimization landscape, leading to more effective policy learning. Our theoretical findings are supported by empirical experiments and provide a principled justification for the online DPO framework in practice.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 58 of its 79 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

58 of 79 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization20 Mar
  2. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization24 Feb
  3. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization7 Nov
  4. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithms15 Oct
  5. 2024The Crucial Role of Samplers in Online Direct Preference Optimization29 Sep
  6. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  7. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  8. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing8 Jun
  9. 2024Filtered Direct Preference Optimization22 Apr
  10. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  11. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model28 Mar
  12. 2024sDPO: Don't Use Your Data All at Once28 Mar
10 more
  1. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
  2. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  3. 2024Reformatted Alignment19 Feb
  4. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  5. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  6. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  7. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2023LIMA: Less Is More for Alignment18 May
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 36

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2015Trust Region Policy Optimizationcited by 2,580
24 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  4. 2024Self-Rewarding Language Modelscited by 681
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  7. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  9. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  10. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  11. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  12. 2024Iterative Reasoning Preference Optimizationcited by 246
  13. 2022Defining and Characterizing Reward Hackingcited by 203
  14. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  15. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  16. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  17. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  18. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  19. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  20. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 79
  21. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  22. 2023Instruction Mining: Instruction Data Selection for Tuning Large Language Modelscited by 58
  23. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 20
  24. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamicscited by 2
  2. 2026Which Pairs to Compare for LLM Post-Training?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Why DPO is a Misspecified Estimator and How to Fix It
  2. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  3. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains
  4. 2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization
  5. 2025What Matters in Data for DPO?
  6. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
  7. 2025Beyond Reward Margin: Rethinking and Resolving Likelihood Displacement in Diffusion Models via Video Generation
  8. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs