mlsanityData to 7 Oct 2026

How Well Can Preference Optimization Generalize Under Noisy Feedback?

arXiv 2510.01458 ↗First posted 1 Oct 2025cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

As large language models (LLMs) advance their capabilities, aligning these models with human preferences has become crucial. Preference optimization, which trains models to distinguish between preferred and non-preferred responses based on human feedback, has become a crucial component for aligning LLMs. However, most existing works assume noise-free feedback, which is unrealistic due to the inherent errors and inconsistencies in human judgments. This paper addresses the impact of noisy feedback on preference optimization, providing generalization guarantees under these conditions. In particular, we consider noise models that correspond to common real-world sources of noise, such as mislabeling and uncertainty. Unlike traditional analyses that assume convergence, our work focuses on finite-step preference optimization, offering new insights that are more aligned with practical LLM training. We describe how generalization decays with different types of noise across levels of noise rates based on the preference data distribution and number of samples. Our analysis for noisy preference learning applies to a broad family of preference optimization losses such as DPO, IPO, SLiC, etc. Empirical validation on contemporary LLMs confirms the practical relevance of our findings, offering valuable insights for developing AI systems that align with human preferences.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 75 of its 91 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

75 of 91 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment28 Sep
  2. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  3. 2024Robust Preference Optimization through Reward Model Distillation29 May
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  6. 2024Token-level Direct Preference Optimization18 Apr
  7. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  8. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  9. 2024Understanding the Learning Dynamics of Alignment with Human Feedback27 Mar
  10. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  11. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  12. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
9 more
  1. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  2. 2024Active Preference Learning for Large Language Models12 Feb
  3. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  4. 2024Decoding-time Realignment of Language Models5 Feb
  5. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback21 Jan
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  8. 2023Preference Ranking Optimization for Human Alignment30 Jun
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 54

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2019Root Mean Square Layer Normalizationcited by 1,425
  12. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
38 more
  1. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  7. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  11. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  13. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  14. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  15. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  16. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  17. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  18. 2021Unsolved Problems in ML Safetycited by 291
  19. 2023AI Alignment: A Comprehensive Surveycited by 256
  20. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  21. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  22. 2023Nash Learning from Human Feedbackcited by 220
  23. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  24. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  25. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  26. 2023Model evaluation for extreme riskscited by 167
  27. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  28. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  29. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  30. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  31. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  32. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  33. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  34. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  35. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  36. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  37. 2023Characterizing Manipulation from AI Systemscited by 43
  38. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedbackcited by 3
  2. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  3. 2026Positive-Unlabeled Preference Optimization For Chest X-ray Report Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  2. 2025One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware and Multi-Source Noise
  3. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
  4. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  5. 2026Unbiased Alignment for Large Language Models with Noisy Preferences
  6. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  7. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  8. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness