mlsanityData to 7 Oct 2026

Attention Limited Reward Learning

arXiv 2607.04590 ↗First posted 6 Jul 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Pairwise human comparisons are a primary interface through which modern AI systems learn human preferences. RLHF and related alignment pipelines typically model such comparisons with Bradley--Terry log-odds, where choice probabilities are governed by latent reward differences. This paper examines what this assumption misses through a reduced-form model motivated by rational inattention, in which each label is generated by a low-capacity evaluation channel. The model separates two forms of ambiguity that standard reward modeling tends to conflate: a comparison may be difficult because the two candidates are genuinely close in value, or because the relevant distinction is hard to detect under limited attention. We show that limited attention can fundamentally distort what pairwise comparisons reveal. In particular, passive comparison data cannot generally distinguish reward, attention, and default tendencies, and heterogeneous attention can make standard Bradley--Terry reward modeling recover misleading rankings. Our analysis shows that learning is governed not by the raw number of labels, but by the amount of attended information each label carries. A case study on human votes over language-model pairs from Chatbot Arena exhibits the predicted signature, a cyclic component of the comparison data that exceeds sampling noise and that no scalar reward can represent; a second case study on perceptual comparisons shows that response times and gaze carry gap information that the labels do not. This perspective suggests that human feedback should be treated not as direct revealed preference, but as an attention-limited measurement process: a weak preference signal may reflect hidden evaluation difficulty rather than genuine indifference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 37 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

22 of 37 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  2. 2023Nash Learning from Human Feedback1 Dec
  3. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  9. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2018AI safety via debatecited by 384
  7. 2016Cooperative Inverse Reinforcement Learningcited by 298
  8. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  9. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  10. 2015Simple, Robust and Optimal Ranking from Pairwise Comparisonscited by 68
  11. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  12. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
  2. 2026Aligning to Illusions: Choice Blindness in Human and AI Feedback
  3. 2023A density estimation perspective on learning from pairwise human preferences
  4. 2025Capturing Individual Human Preferences with Reward Features
  5. 2025Reward Model Interpretability via Optimal and Pessimal Tokens
  6. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferences
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2025Reviving The Classics: Active Reward Modeling in Large Language Model Alignment