mlsanityData to 14 Oct 2026

A Descriptive and Normative Theory of Human Beliefs in RLHF

arXiv 2506.01692 ↗First posted 2 Jun 2025cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Human preferences in RLHF are typically modeled as a function of the human's reward function or corresponding optimal state-action values. In this work, we propose that human beliefs about the capabilities of the agent being trained also play a key role in preference generation. We examine two questions related to this hypothesis, one descriptive and one normative, respectively: Do human labelers' beliefs about agent capabilities affect the preferences that they provide? And what is the ideal set of beliefs about an agent -- and resulting preferences -- for humans to have? We propose a new preference model that incorporates human beliefs and provide a normative theory that bounds the error on the final learned policy based on the mismatch between the human's beliefs and an idealized set of beliefs. We then confirm via a human study that beliefs about agent capabilities do, in fact, significantly affect preferences and can be influenced through simple interventions. Additionally, we empirically show through synthetic experiments that it is often suboptimal for human preference labelers to assume agent optimality. Collectively, these results theoretically and empirically demonstrate how reducing the mismatch between human beliefs and agent capabilities can lead to more performant RLHF and point toward new best practices for RLHF practitioners.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 38 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

17 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Nash Learning from Human Feedback1 Dec
  2. 2023Is RLHF More Difficult than Standard RL?25 Jun
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  5. 2022Models of human preference for learning reward functions5 Jun
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 10

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  4. 2016Concrete Problems in AI Safetycited by 1,642
  5. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  6. 2023AI Alignment: A Comprehensive Surveycited by 263
  7. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  8. 2024Reinforcement learningcited by 247
  9. 2018Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behaviorcited by 40
  10. 2021Human irrationality: both bad and good for reward inferencecited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Influencing Humans to Conform to Preference Models for RLHF
  2. 2024Beyond Preferences in AI Alignment
  3. 2024Mapping Social Choice Theory to RLHF
  4. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory
  5. 2023The History and Risks of Reinforcement Learning and Human Feedback
  6. 2026Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction
  7. 2026RLHF May Not Reflect Genuine Preferences
  8. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications