mlsanityData to 7 Oct 2026

When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF

arXiv 2512.00709 ↗First posted 30 Nov 2025cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Quality of datasets plays an important role in large language model (LLM) alignment. In collecting human feedback, however, preference flipping is ubiquitous and causes corruption in data annotation; the issue necessitates the alignment algorithms with improved robustness against potential flipped pairs. To this end, this paper introduces a Flipping-Aware Direct Preference Optimization (FA-DPO) algorithm tailored to preference flipping from a reinforcement learning with human feedback (RLHF) perspective. We dissect the inherent human intention model and the preference flipping mechanism introduced by external factors as two distinct stages; in the latter, we introduce an instance-dependent flipping probability on the basis of the Bradley-Terry (BT) model. Further, by leveraging features relevant to preference annotation, we capture uncertainty in judgments and model preference flipping patterns. In practice, we design a simple yet efficient iterative optimization algorithm compatible with the original RLHF and DPO algorithms. In our experiments, we investigate the instance-dependent preference flipping model under multiple circumstances for evaluation of our proposed method, as well as other baseline methods.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 51 references that are in the map. First posted on arXiv 30 Nov 2025.

Builds on

33 of 51 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025ComPO: Preference Alignment via Comparison Oracles8 May
  2. 2024Robust Preference Optimization through Reward Model Distillation29 May
  3. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization26 May
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  6. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  10. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
12 more
  1. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  2. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  3. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  4. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  5. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  6. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  7. 2021FINE Samples for Learning with Noisy Labelscited by 82
  8. 2021Sample Selection with Uncertainty of Losses for Learning with Noisy Labelscited by 79
  9. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemblecited by 43
  10. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  11. 2022Identifiability of Label Noise Transition Matrixcited by 34
  12. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedbackcited by 0
  2. 2026Aligning to Illusions: Choice Blindness in Human and AI Feedbackcited by 0
  3. 2026Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  2. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  3. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  4. 2025Doubly Robust Alignment for Large Language Models
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  6. 2025Online Preference Alignment for Language Models via Count-based Exploration
  7. 2024Investigating on RLHF methodology
  8. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration