mlsanityData to 7 Oct 2026

Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples

arXiv 2502.09650 ↗First posted 11 Feb 2025cs.CL cs.AI cs.LGcited by 37

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The alignment of large language models (LLMs) often assumes that using more clean data yields better outcomes, overlooking the match between model capacity and example difficulty. Challenging this, we propose a new principle: Preference data vary in difficulty, and overly difficult examples hinder alignment, by exceeding the model's capacity. Through systematic experimentation, we validate this principle with three key findings: (1) preference examples vary in difficulty, as evidenced by consistent learning orders across alignment runs; (2) overly difficult examples significantly degrade performance across four LLMs and two datasets; and (3) the capacity of a model dictates its threshold for handling difficult examples, underscoring a critical relationship between data selection and model capacity. Building on this principle, we introduce Selective DPO, which filters out overly difficult examples. This simple adjustment improves alignment performance by 9-16% in win rates on the AlpacaEval 2 benchmark compared to the DPO baseline, suppressing a series of DPO variants with different algorithmic adjustments. Together, these results illuminate the importance of aligning data difficulty with model capacity, offering a transformative perspective for improving alignment strategies in LLMs. Code is available at https://github.com/glorgao/SelectiveDPO.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 67 of its 90 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

67 of 90 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Rethinking Data Selection at Scale: Random Selection is Almost All You Need12 Oct
  2. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  3. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models4 Aug
  4. 2024WPO: Enhancing RLHF with Weighted Preference Optimization17 Jun
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  7. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  8. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  9. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
  10. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  11. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  12. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
4 more
  1. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2022Training language models to follow instructions with human feedback4 Mar
  4. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Mistral 7Bcited by 3,065
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  12. 2021Ethical and social risks of harm from Language Modelscited by 957
38 more
  1. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  2. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  3. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  5. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  6. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  8. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  9. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  10. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  12. 2024A Survey on Data Selection for Language Modelscited by 230
  13. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 201
  14. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  15. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  16. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  17. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  18. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  19. 2014Learning to Executecited by 105
  20. 2020When Do Curricula Work?cited by 105
  21. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  22. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  23. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  24. 2019Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narrativescited by 59
  25. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  26. 2017How hard can it be? Estimating the difficulty of visual search in an imagecited by 52
  27. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  28. 2024Curriculum Direct Preference Optimization for Diffusion and Consistency Modelscited by 36
  29. 2021Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learningcited by 34
  30. 2023Rethinking the Instruction Quality: LIFT is What You Needcited by 33
  31. 2021Curriculum learning for language modelingcited by 32
  32. 2024Understanding Finetuning for Factual Knowledge Extractioncited by 31
  33. 2024Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learningcited by 28
  34. 2024Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrappingcited by 14
  35. 2023Reverse Stable Diffusion: What prompt was used to generate this image?cited by 7
  36. 2024Curriculum Learning with Quality-Driven Data Selectioncited by 7
  37. 2022Exploring the Learning Difficulty of Data Theory and Measurecited by 4
  38. 2024Does the Definition of Difficulty Matter? Scoring Functions and their Role for Curriculum Learningcited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 18.

37 papers.

Sort:

By area:

  1. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  2. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedbackcited by 30
  3. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  4. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 24
  5. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
  6. 2025Enhancing Jailbreak Attacks on LLMs via Persona Promptscited by 10
  7. 2025Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspectivecited by 7
  8. 2025Difficulty-Based Preference Data Selection by DPO Implicit Reward Gapcited by 7
  9. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  10. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Modelcited by 6
  11. 2025Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuningcited by 6
  12. 2025CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationalecited by 5
25 more
  1. 2025Less is More: Improving LLM Alignment via Preference Data Selectioncited by 4
  2. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
  3. 2026FastLightGen: Fast and Light Video Generation with Fewer Steps and Parameterscited by 4
  4. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimizationcited by 3
  5. 2025DRIVE: Data Curation Best Practices for Reinforcement Learning with Verifiable Reward in Competitive Code Generationcited by 3
  6. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 3
  7. 2026MARTI-MARS2: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generationcited by 3
  8. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignmentcited by 2
  9. 2026Small-Margin Preferences Still Matter-If You Train Them Rightcited by 2
  10. 2025Data Selection for LLM Alignment Using Fine-Grained Preferencescited by 1
  11. 2025Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMscited by 1
  12. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  13. 2026Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understandingcited by 1
  14. 2026DOG-DPO:Dynamic Optimization in Geometry for Safety Alignmentcited by 1
  15. 2026Capacity-Dependent Effects of Data Selection for Reasoningcited by 1
  16. 2025Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficultycited by 0
  17. 2026ZPD Detector: Data Selection via Capability-Difficulty Alignment for Large Language Modelscited by 0
  18. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferencescited by 0
  19. 2026DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselinescited by 0
  20. 2026The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoningcited by 0
  21. 2026BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciencescited by 0
  22. 2026Wnuan: Staged Post-Training for Question Answering over Proprietary Enterprise Knowledgecited by 0
  23. 2026CalcSeg: Confidence-aware 3D Latent Context Curriculum Learning For Myocardial Scar Segmentation From Single-Stack LGE-CMRscited by 0
  24. 2026PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimizationcited by 0
  25. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
  3. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  4. 2024Efficient Alignment of Large Language Models via Data Sampling
  5. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  6. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  7. 2024Aligner: Efficient Alignment by Learning to Correct
  8. 2025CM-Align: Consistency-based Multilingual Alignment for Large Language Models