mlsanityData to 14 Oct 2026

Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap

arXiv 2508.04149 ↗First posted 6 Aug 2025cs.CL cs.AI cs.LGcited by 7

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human preferences is a critical challenge in AI research. While methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are widely used, they often rely on large, costly preference datasets. The current work lacks methods for high-quality data selection specifically for preference data. In this work, we introduce a novel difficulty-based data selection strategy for preference datasets, grounded in the DPO implicit reward mechanism. By selecting preference data examples with smaller DPO implicit reward gaps, which are indicative of more challenging cases, we improve data efficiency and model alignment. Our approach consistently outperforms five strong baselines across multiple datasets and alignment tasks, achieving superior performance with only 10% of the original data. This principled, efficient selection method offers a promising solution for scaling LLM alignment with limited resources.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 60 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

39 of 60 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples11 Feb
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Token-level Direct Preference Optimization18 Apr
  4. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  5. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2023LIMA: Less Is More for Alignment18 May
  8. 2022Training language models to follow instructions with human feedback4 Mar
  9. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
18 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 648
  2. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  3. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  5. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  6. 2020Artificial Intelligence, Values and Alignmentcited by 448
  7. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  9. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  10. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  12. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  13. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
  14. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  15. 2025Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolutioncited by 116
  16. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  17. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 44
  18. 2021Self-Guided Curriculum Learning for Neural Machine Translationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selectioncited by 2
  2. 2026DataProphet: Demystifying Supervision Data Generalization in Multimodal LLMscited by 1
  3. 2026Smaller Models, Better Rejects: Preference Distillation Scalingcited by 1
  4. 2025Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficultycited by 0
  5. 2026Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Datacited by 0
  6. 2026Data Difficulty and the Generalization--Extrapolation Tradeoff in LLM Fine-Tuningcited by 0
  7. 2026PoEM: Predicting RL Outcomes from Existing Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  2. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  3. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  4. 2025A Survey of Direct Preference Optimization
  5. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  6. 2024Bootstrapping Language Models with DPO Implicit Rewards
  7. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  8. 2025Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm