mlsanityData to 7 Oct 2026

AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection

arXiv 2609.05899 ↗First posted 5 Sep 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models with human preferences remains a challenge, primarily due to the critical role of preference data quality in effective alignment. Existing datasets are frequently plagued by inherent noise and distribution shifts, which inherently limit model performance. To bridge this gap, we propose AlignDiff, a preference data filtering framework driven by intrinsic model signals. AlignDiff first identifies samples with clear preferences using both positive and inverse signals, then prioritizes the more challenging samples based on the average negative log-likelihood gap, encouraging the model to learn richer information from them. AlignDiff is evaluated on two widely used model families (LLaMA and Qwen) and three benchmarks widely adopted in the alignment community (AlpacaEval 2.0, Arena-Hard, and MT-Bench). Across all settings, it consistently outperforms seven strong baselines. We conduct comprehensive ablation studies to validate the effectiveness of AlignDiff, and further show that difficulty-based curriculum learning improves model performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 62 references that are in the map. First posted on arXiv 5 Sep 2026.

Builds on

41 of 62 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge27 Jul
  2. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms11 Jun
  3. 2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset4 Apr
  4. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization20 Mar
  5. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?25 Feb
  6. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples11 Feb
  7. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  8. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  9. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment6 Jun
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  11. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  12. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
7 more
  1. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  2. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  3. 2023On Diversified Preferences of Large Language Model Alignment12 Dec
  4. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2022Training language models to follow instructions with human feedback4 Mar
  7. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  10. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  11. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  12. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
10 more
  1. 2024Rethinking Data Selection for Supervised Fine-Tuningcited by 35
  2. 2024Question Calibration and Multi-Hop Modeling for Temporal Question Answeringcited by 25
  3. 2023Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflictscited by 21
  4. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparisoncited by 20
  5. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 14
  6. 2026BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluationcited by 12
  7. 2025StructCoh: Structured Contrastive Learning for Context-Aware Text Semantic Matchingcited by 8
  8. 2024Towards Comprehensive Preference Data Collection for Reward Modelingcited by 5
  9. 2025Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modelingcited by 4
  10. 2026Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneckcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignment
  2. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  3. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  4. 2025Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation
  5. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  6. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
  7. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignment
  8. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization