mlsanityData to 14 Oct 2026

Impact of Preference Noise on the Alignment Performance of Generative Language Models

arXiv 2404.09824 ↗First posted 15 Apr 2024cs.CLcited by 41

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

A key requirement in developing Generative Language Models (GLMs) is to have their values aligned with human values. Preference-based alignment is a widely used paradigm for this purpose, in which preferences over generation pairs are first elicited from human annotators or AI systems, and then fed into some alignment techniques, e.g., Direct Preference Optimization. However, a substantial percent (20 - 40%) of the preference pairs used in GLM alignment are noisy, and it remains unclear how the noise affects the alignment performance and how to mitigate its negative impact. In this paper, we propose a framework to inject desirable amounts and types of noise to the preferences, and systematically study the impact of preference noise on the alignment performance in two tasks (summarization and dialogue generation). We find that the alignment performance can be highly sensitive to the noise rates in the preference data: e.g., a 10 percentage points (pp) increase of the noise rate can lead to 30 pp drop in the alignment performance (in win rate). To mitigate the impact of noise, confidence-based data filtering shows significant benefit when certain types of noise are present. We hope our work can help the community better understand and mitigate the impact of preference noise in GLM alignment.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 43 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

33 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
18 more
  1. 2024Self-Rewarding Language Modelscited by 681
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  3. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  6. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  7. 2023AI Alignment: A Comprehensive Surveycited by 263
  8. 2023Nash Learning from Human Feedbackcited by 224
  9. 2020A Survey of Label-noise Representation Learning: Past, Present and Futurecited by 122
  10. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  11. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  12. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 77
  13. 2021Classifier Calibration: A survey on how to assess and improve predicted class probabilitiescited by 73
  14. 2022Training Uncertainty-Aware Classifiers with Conformalized Deep Learningcited by 64
  15. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  16. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  17. 2019Preference-based Interactive Multi-Document Summarisationcited by 11
  18. 2023It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 23; 2026: 11.

41 papers.

Sort:

By area:

  1. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  2. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  3. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  4. 2025A Survey of Direct Preference Optimizationcited by 36
  5. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31
  6. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 25
  7. 2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
  8. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 17
  9. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translationcited by 8
  10. 2025Difficulty-Based Preference Data Selection by DPO Implicit Reward Gapcited by 7
  11. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 6
  12. 2024Self-Steering Optimization: Autonomous Preference Optimization for Large Language Modelscited by 5
29 more
  1. 2025Iterative Label Refinement Matters More than Preference Optimization under Weak Supervisioncited by 5
  2. 2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modelingcited by 5
  3. 2025Dynamic Noise Preference Optimization: Self-Improvement of Large Language Models with Self-Synthetic Datacited by 4
  4. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  5. 2025On Symmetric Losses for Robust Policy Optimization with Noisy Preferencescited by 4
  6. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 4
  7. 2026GNN-as-Judge: Unleashing the Power of LLMs for Graph Learning with GNN Feedbackcited by 4
  8. 2024The Real, the Better: Aligning Large Language Models with Online Human Behaviorscited by 3
  9. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
  10. 2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHFcited by 3
  11. 2025Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratchcited by 2
  12. 2025One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware and Multi-Source Noisecited by 2
  13. 2025Influence Functions for Preference Dataset Pruningcited by 2
  14. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
  15. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignmentcited by 2
  16. 2026Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testingcited by 2
  17. 2025When Preferences Diverge: Aligning Diffusion Models with Minority-Aware Adaptive DPOcited by 1
  18. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  19. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learningcited by 1
  20. 2026Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Modelscited by 1
  21. 2026Scaling Reward Modeling without Human Supervisioncited by 1
  22. 2025Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learningcited by 0
  23. 2025Incentivizing High-Quality Human Annotations with Golden Questionscited by 0
  24. 2025Users as Annotators: LLM Preference Learning from Comparison Modecited by 0
  25. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Strongercited by 0
  26. 2026Active-GRPO: Adaptive Imitation and Self-Improving Reasoning for Molecular Optimizationcited by 0
  27. 2026Unbiased Alignment for Large Language Models with Noisy Preferencescited by 0
  28. 2026Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labelscited by 0
  29. 2026ABC-Align: Prediction-Powered Alignment with Adaptive Bias Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  2. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  3. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  4. 2024Negative-Prompt-driven Alignment for Generative Language Model
  5. 2025ComPO: Preference Alignment via Comparison Oracles
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
  8. 2023On Diversified Preferences of Large Language Model Alignment