mlsanityData to 14 Oct 2026

Towards Understanding Valuable Preference Data for Large Language Model Alignment

arXiv 2510.13212 ↗First posted 15 Oct 2025cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language model (LLM) alignment is typically achieved through learning from human preference comparisons, making the quality of preference data critical to its success. Existing studies often pre-process raw training datasets to identify valuable preference pairs using external reward models or off-the-shelf LLMs, achieving improved overall performance but rarely examining whether individual, selected data point is genuinely beneficial. We assess data quality through individual influence on validation data using our newly proposed truncated influence function (TIF), which mitigates the over-scoring present in traditional measures and reveals that preference data quality is inherently a property of the model. In other words, a data pair that benefits one model may harm another. This leaves the need to improve the preference data selection approaches to be adapting to specific models. To this end, we introduce two candidate scoring functions (SFs) that are computationally simpler than TIF and positively correlated with it. They are also model dependent and can serve as potential indicators of individual data quality for preference data selection. Furthermore, we observe that these SFs inherently exhibit errors when compared to TIF. To this end, we combine them to offset their diverse error sources, resulting in a simple yet effective data selection rule that enables the models to achieve a more precise selection of valuable preference data. We conduct experiments across diverse alignment benchmarks and various LLM families, with results demonstrating that better alignment performance can be achieved using less data, showing the generality of our findings and new methods. Our code is publicly available at https://github.com/tmlr-group/TIF_LossDiff-IRM.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 72 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

49 of 72 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?25 Feb
  2. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples11 Feb
  3. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models4 Aug
  4. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Filtered Direct Preference Optimization22 Apr
  7. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
  8. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models15 Feb
  9. 2024Active Preference Learning for Large Language Models12 Feb
  10. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  11. 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference5 Dec
  12. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
3 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr

Other references · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  11. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
22 more
  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  3. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  6. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  7. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  8. 2024A Survey on Data Selection for Language Modelscited by 236
  9. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
  10. 2024Reinforcement Learning for LLM Post-Training: A Surveycited by 92
  11. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 79
  12. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyondcited by 61
  13. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  14. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 41
  15. 2025Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuningcited by 39
  16. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  17. 2025Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Modelscited by 25
  18. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 23
  19. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 23
  20. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparisoncited by 21
  21. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forgetcited by 14
  22. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025LLM Unlearning with LLM Beliefscited by 17
  2. 2026Explainable LLM Unlearning Through Reasoningcited by 7
  3. 2026What Does Preference Learning Recover from Pairwise Comparison Data?cited by 3
  4. 2026Let the Target Select for Itself: Data Selection via Target-Aligned Pathscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  3. 2023On Diversified Preferences of Large Language Model Alignment
  4. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection
  5. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  6. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  7. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  8. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective