mlsanityData to 14 Oct 2026

Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization

arXiv 2506.17252 ↗First posted 8 Jun 2025cs.LG cs.AIcited by 17

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its performance is highly dependent on the quality of the underlying human preference data. To address this bottleneck, prior work has explored various data selection strategies, but these methods often overlook the impact of the evolving states of the language model during the optimization process. In this paper, we introduce a novel problem: Sample Scheduling for DPO, which aims to dynamically and adaptively schedule training samples based on the model's evolving batch-wise states throughout preference optimization. To solve this problem, we propose SamS, an efficient and effective algorithm that adaptively selects samples in each training batch based on the LLM's learning feedback to maximize the potential generalization performance. Notably, without modifying the core DPO algorithm, simply integrating SamS significantly improves performance across tasks, with minimal additional computational overhead. This work points to a promising new direction for improving LLM alignment through batch-wise sample selection, with potential generalization to RLHF and broader supervised learning paradigms.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 84 of its 107 references that are in the map. First posted on arXiv 8 Jun 2025.

Builds on

84 of 107 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples11 Feb
  2. 2024Sample-Efficient Alignment for LLMs3 Nov
  3. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization5 Sep
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  6. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  7. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization31 Mar
  8. 2024sDPO: Don't Use Your Data All at Once28 Mar
  9. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  10. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  11. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  12. 2024Direct Preference Optimization with an Offset16 Feb
9 more
  1. 2024Active Preference Learning for Large Language Models12 Feb
  2. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  3. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  4. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  7. 2023Preference Ranking Optimization for Human Alignment30 Jun
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 63

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
38 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  6. 2024Self-Rewarding Language Modelscited by 681
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  8. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  10. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  12. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  13. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  14. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  15. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  16. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  17. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  18. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  19. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  20. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  21. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  22. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  23. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  24. 2023Pretraining Language Models with Human Preferencescited by 247
  25. 2023Fine-tuning Language Models for Factualitycited by 221
  26. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  27. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  28. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  29. 2020Neural Thompson Samplingcited by 128
  30. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  31. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 103
  32. 2015Contextual Dueling Banditscited by 101
  33. 2023HIPTrack: Visual Tracking with Historical Promptscited by 71
  34. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  35. 2021EE-Net: Exploitation-Exploration Neural Networks in Contextual Banditscited by 44
  36. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 43
  37. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
  38. 2022SparseTT: Visual Tracking with Sparse Transformerscited by 36

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 14.

17 papers.

Sort:

By area:

  1. 2026Your Group-Relative Advantage Is Biasedcited by 34
  2. 2026Does Your Reasoning Model Implicitly Know When to Stop Thinking?cited by 19
  3. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 15
  4. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedbackcited by 11
  5. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 11
  6. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaborationcited by 10
  7. 2025LD-RPS: Zero-Shot Unified Image Restoration via Latent Diffusion Recurrent Posterior Samplingcited by 9
  8. 2026Start Classifying: Categorical Critics for LLM Reinforcement Learningcited by 3
  9. 2025What Is Preference Optimization Doing, and Why?cited by 2
  10. 2026RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimizationcited by 2
  11. 2026Uni-MDTrack: Learning Decoupled Memory and Dynamic States for Parameter-Efficient Visual Tracking in All Modalitycited by 1
  12. 2026SynGR: Unleashing the Potential of Cross-Modal Synergy for Generative Recommendationcited by 1
5 more
  1. 2026Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMscited by 1
  2. 2026Federated Reasoning Distillation Framework with Model Learnability-Aware Data Allocationcited by 0
  3. 2026Models as Tools: An Agentic Coordination Framework for Unified Multimodal Visual Trackingcited by 0
  4. 2026Se-DPO: Self-Evolving Token Credit for Direct Preference Optimizationcited by 0
  5. 2026SquidAgent: Parallelize Wisely, Coordinate Efficientlycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  2. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  3. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  4. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  5. 2024Multi-Reference Preference Optimization for Large Language Models
  6. 2024The Crucial Role of Samplers in Online Direct Preference Optimization
  7. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  8. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts