# Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?

arXiv: https://arxiv.org/abs/2506.19467 · HTML: https://mlsanity.com/paper/2506.19467/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 25 of its 66 references that are in the map.
- First posted on arXiv 24 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [HelpSteer2-Preference: Complementing Ratings with Preferences](https://mlsanity.com/paper/2410.01257/) (2024)
- [Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse](https://mlsanity.com/paper/2410.21333/) (2024)
- [Automated Annotation with Generative AI Requires Validation](https://mlsanity.com/paper/2306.00176/) (2023)
- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- [Reasoning Models Better Express Their Confidence](https://mlsanity.com/paper/2505.14489/) (2025)
- [Benchmarking Distributional Alignment of Large Language Models](https://mlsanity.com/paper/2411.05403/) (2024)
- [LEXam: Benchmarking Legal Reasoning on 340 Law Exams](https://mlsanity.com/paper/2505.12864/) (2025)
- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [Best Practices for Text Annotation with Large Language Models](https://mlsanity.com/paper/2402.05129/) (2024)
- [CREAM: Consistency Regularized Self-Rewarding Language Models](https://mlsanity.com/paper/2410.12735/) (2024)
- [Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions](https://mlsanity.com/paper/2502.20897/) (2025)
- [Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models](https://mlsanity.com/paper/2502.08922/) (2025)
- [Synthesizing Public Opinions with LLMs: Role Creation, Impacts, and the Future to eDemorcacy](https://mlsanity.com/paper/2504.00241/) (2025)

## Built on by (most cited first, 10 of 10)

- [Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels](https://mlsanity.com/paper/2605.29800/) (2026)
- [HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants](https://mlsanity.com/paper/2509.08494/) (2025)
- [Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective](https://mlsanity.com/paper/2601.03154/) (2026)
- [LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps](https://mlsanity.com/paper/2604.27345/) (2026)
- [STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems](https://mlsanity.com/paper/2605.02122/) (2026)
- [ORCA: Open-ended Response Correctness Assessment for Audio Question Answering](https://mlsanity.com/paper/2512.09066/) (2025)
- [Learning Who Disagrees: Demographic Importance Weighting for Modeling Annotator Distributions with DiADEM](https://mlsanity.com/paper/2604.08425/) (2026)
- [Triviality Corrected Endogenous Reward](https://mlsanity.com/paper/2604.11522/) (2026)
- [EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks](https://mlsanity.com/paper/2608.04549/) (2026)
- [Decomposing and Predicting Annotation Disagreement in Harmful Content Across Languages](https://mlsanity.com/paper/2610.18588/) (2026)

## Abstract (arXiv)

Variation in human annotation (i.e., disagreements) is common in NLP, often reflecting important information like task subjectivity and sample ambiguity. Modeling this variation is important for applications that are sensitive to such information. Although RLVR-style reasoning (Reinforcement Learning with Verifiable Rewards) has improved Large Language Model (LLM) performance on many tasks, it remains unclear whether such reasoning enables LLMs to capture informative variation in human annotation. In this work, we evaluate the influence of different reasoning settings on LLM disagreement modeling. We systematically evaluate each reasoning setting across model sizes, distribution expression methods, and steering methods, resulting in 60 experimental setups across 3 tasks. Surprisingly, our results show that RLVR-style reasoning degrades performance in disagreement modeling, while naive Chain-of-Thought (CoT) reasoning improves the performance of RLHF LLMs (RL from human feedback). These findings underscore the potential risk of replacing human annotators with reasoning LLMs, especially when disagreements are important.