mlsanityData to 7 Oct 2026

Diverging Preferences: When do Annotators Disagree and do Models Know?

arXiv 2410.14632 ↗First posted 18 Oct 2024cs.CLcited by 47

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

We examine diverging preferences in human-labeled preference datasets. We develop a taxonomy of disagreement sources spanning ten categories across four high-level classes and find that the majority of disagreements are due to factors such as task underspecification or response style. Our findings challenge a standard assumption in reward modeling methods that annotator disagreements can be attributed to simple noise. We then explore how these findings impact two areas of LLM development: reward modeling training and evaluation. In our experiments, we demonstrate how standard reward modeling (e.g., Bradley-Terry) and LLM-as-Judge evaluation methods fail to account for divergence between annotators. These findings highlight challenges in LLM evaluations, which are greatly influenced by divisive features like response style, and in developing pluralistically aligned LLMs. To address these issues, we develop methods for identifying diverging preferences to mitigate their influence in evaluations and during LLM training.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 56 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

31 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Improving Context-Aware Preference Modeling for Language Models20 Jul
  2. 2024A Roadmap to Pluralistic Alignment7 Feb
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  12. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
16 more
  1. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  3. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  4. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 253
  5. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  6. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  7. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  8. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  9. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  10. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  11. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 93
  12. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  13. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  14. 2024The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labelscited by 37
  15. 2022Investigating Reasons for Disagreement in Natural Language Inferencecited by 29
  16. 2023Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Designcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 23; 2026: 21.

47 papers.

Sort:

By area:

  1. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  2. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
  3. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  4. 2025Value Profiles for Encoding Human Variationcited by 29
  5. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 24
  6. 2024VisionArena: 230K Real World User-VLM Conversations with Preference Labelscited by 22
  7. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  8. 2025AI Alignment at Your Discretioncited by 20
  9. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  10. 2025User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signalcited by 19
  11. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  12. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
35 more
  1. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  2. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 11
  3. 2025Meaning Is Not A Metric: Using LLMs to make cultural context legible at scalecited by 10
  4. 2026Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLPcited by 10
  5. 2025SafeSteer: Interpretable Safety Steering with Refusal-Evasion in LLMscited by 9
  6. 2025Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?cited by 9
  7. 2025CoPL: Collaborative Preference Learning for Personalizing LLMscited by 8
  8. 2025DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimizationcited by 7
  9. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 6
  10. 2026PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harmcited by 5
  11. 2026One Size, Many Fits: Aligning Diverse Group-Wise Click Preferences in Large-Scale Advertising Image Generationcited by 4
  12. 2025From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLPcited by 3
  13. 2024From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Setcited by 2
  14. 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modelingcited by 2
  15. 2025Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobscited by 1
  16. 2026Robust AI Evaluation through Maximal Lotteriescited by 1
  17. 2026Personalized Group Relative Policy Optimization for Heterogenous Preference Alignmentcited by 1
  18. 2026RLHF May Not Reflect Genuine Preferencescited by 1
  19. 2025ZEBRA: Leveraging Model-Behavioral Knowledge for Zero-Annotation Preference Dataset Constructioncited by 0
  20. 2025VAL-Bench: Belief Consistency as a measure for Value Alignment in Language Modelscited by 0
  21. 2025Tracing How Annotators Think: Augmenting Preference Judgments with Reading Processescited by 0
  22. 2026Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedbackcited by 0
  23. 2026Scaling Reward Modeling without Human Supervisioncited by 0
  24. 2026DARC: Disagreement-Aware Alignment via Risk-Constrained Decodingcited by 0
  25. 2026Aligning to Illusions: Choice Blindness in Human and AI Feedbackcited by 0
  26. 2026Variance-aware Reward Modeling with Anchor Guidancecited by 0
  27. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approachcited by 0
  28. 2026Coherence Maximization Improves Pluralistic Alignmentcited by 0
  29. 2026AI Alignment From Social Choice Perspectivescited by 0
  30. 2026Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrastcited by 0
  31. 2026PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimizationcited by 0
  32. 2026PlurVA-LLM-2026 Shared Task Track-1: Pluralistic Value Alignment in LLMs via Multilingual Fine-Tuning and Threshold Calibrationcited by 0
  33. 2026Diffusion Reward Modelscited by 0
  34. 2026ABC-Align: Prediction-Powered Alignment with Adaptive Bias Controlcited by 0
  35. 2026Uncertainty-Normalized Margins for Direct Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
  2. 2026Automated Concept Discovery for LLM-as-a-Judge Preference Analysis
  3. 2025Detecting Prefix Bias in LLM-based Reward Models
  4. 2024Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets
  5. 2024Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuning
  6. 2025Users as Annotators: LLM Preference Learning from Comparison Mode
  7. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?
  8. 2025Rethinking Human Preference Evaluation of LLM Rationales