mlsanityData to 7 Oct 2026

The Crucial Role of Samplers in Online Direct Preference Optimization

arXiv 2409.19605 ↗First posted 29 Sep 2024cs.LG cs.CLcited by 26

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) has emerged as a stable, scalable, and efficient solution for language model alignment. Despite its empirical success, the optimization properties, particularly the impact of samplers on its convergence rates, remain under-explored. In this paper, we provide a rigorous analysis of DPO's convergence rates with different sampling strategies under the exact gradient setting, revealing a surprising separation: uniform sampling achieves linear convergence, while our proposed online sampler achieves quadratic convergence. We further adapt the sampler to practical settings by incorporating posterior distributions and logit mixing, demonstrating improvements over previous methods. For example, it outperforms vanilla DPO by over 7.4% on Safe-RLHF dataset. Our results not only offer insights into the theoretical understanding of DPO but also pave the way for further algorithm designs.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 35 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

29 of 35 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  2. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models21 Jun
  3. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coverage3 Jun
  4. 2024Preference Learning Algorithms Do Not Learn Preference Rankings29 May
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models15 Feb
  7. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  8. 2024Decoding-time Realignment of Language Models5 Feb
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep

Other references · 18

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  9. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  10. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  11. 2023AI Alignment: A Comprehensive Surveycited by 256
  12. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
6 more
  1. 2023Nash Learning from Human Feedbackcited by 220
  2. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  3. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  4. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  5. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  6. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 10.

26 papers.

Sort:

By area:

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  3. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 23
  4. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  5. 2026Self-Hinting Language Models Enhance Reinforcement Learningcited by 19
  6. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 18
  7. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  8. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  9. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 12
  10. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?cited by 11
  11. 2025ComPO: Preference Alignment via Comparison Oraclescited by 11
  12. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
14 more
  1. 2026Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularizationcited by 10
  2. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  3. 2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learningcited by 5
  4. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  5. 2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 5
  6. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5
  7. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
  8. 2025Leveraging Sparsity for Sample-Efficient Preference Learning: A Theoretical Perspectivecited by 2
  9. 2026Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulationcited by 2
  10. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamicscited by 2
  11. 2026LAD: Learning Advantage Distribution for Reasoningcited by 2
  12. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 1
  13. 2026Efficient Exploration for Iterative Nash Preference Optimizationcited by 0
  14. 2026A Zeroth-Order Paradigm for LLM Preference Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization
  2. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  3. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  4. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  5. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  6. 2025What Matters in Data for DPO?
  7. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
  8. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights