mlsanityData to 14 Oct 2026

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types

arXiv 2208.10687 ↗First posted 23 Aug 2022cs.LG cs.AIcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

When inferring reward functions from human behavior (be it demonstrations, comparisons, physical corrections, or e-stops), it has proven useful to model the human as making noisy-rational choices, with a "rationality coefficient" capturing how much noise or entropy we expect to see in the human behavior. Prior work typically sets the rationality level to a constant value, regardless of the type, or quality, of human feedback. However, in many settings, giving one type of feedback (e.g. a demonstration) may be much more difficult than a different type of feedback (e.g. answering a comparison query). Thus, we expect to see more or less noise depending on the type of human feedback. In this work, we advocate that grounding the rationality coefficient in real data for each feedback type, rather than assuming a default value, has a significant positive effect on reward learning. We test this in both simulated experiments and in a user study with real human feedback. We find that overestimating human rationality can have dire effects on reward learning accuracy and regret. We also find that fitting the rationality coefficient to human data enables better reward learning, even when the human deviates significantly from the noisy-rational choice model due to systematic biases. Further, we find that the rationality level affects the informativeness of each feedback type: surprisingly, demonstrations are not always the most informative -- when the human acts very suboptimally, comparisons actually become more informative, even when the rationality level is the same for both. Ultimately, our results emphasize the importance and advantage of paying attention to the assumed human-rationality level, especially when agents actively learn from multiple types of human feedback.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 51 references that are in the map. First posted on arXiv 23 Aug 2022.

Builds on

20 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 18

  1. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  2. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  3. 2017Inverse Reward Designcited by 249
  4. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
  5. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  6. 2016The Off-Switch Gamecited by 97
  7. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  8. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  9. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  10. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  11. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  12. 2018Probabilistically Safe Robot Planning with Confidence-Based Human Predictionscited by 28
6 more
  1. 2021Human irrationality: both bad and good for reward inferencecited by 25
  2. 2020Quantifying Hypothesis Space Misspecification in Learning from Human-Robot Demonstrations and Physical Correctionscited by 19
  3. 2021Modeling the Mistakes of Boundedly Rational Agents Within a Bayesian Theory of Mindcited by 19
  4. 2017Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learningcited by 17
  5. 2021Situational Confidence Assistance for Lifelong Shared Autonomycited by 15
  6. 2020I Know What You Meant: Learning Human Objectives by (Under)estimating Their Choice Setcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 10; 2025: 3; 2026: 6.

26 papers.

Sort:

By area:

  1. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  2. 2024Beyond Preferences in AI Alignmentcited by 49
  3. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 21
  4. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 21
  5. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedbackcited by 15
  6. 2025Reward Learning from Multiple Feedback Typescited by 9
  7. 2024Pareto-Optimal Learning from Preferences with Hidden Contextcited by 8
  8. 2024The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewardscited by 7
  9. 2023(Ir)rationality in AI: State of the Art, Research Challenges and Open Questionscited by 5
  10. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  11. 2024BCR-DRL: Behavior- and Context-aware Reward for Deep Reinforcement Learning in Human-AI Coordinationcited by 3
  12. 2022Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learningcited by 1
14 more
  1. 2025Model-Based Soft Maximization of Suitable Metrics of Long-Term Human Powercited by 1
  2. 2026Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learningcited by 1
  3. 2023Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?cited by 0
  4. 2023On the Effective Horizon of Inverse Reinforcement Learningcited by 0
  5. 2024Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrationscited by 0
  6. 2024On the Effect of Robot Errors on Human Teaching Dynamicscited by 0
  7. 2024Extended Reality System for Robotic Learning from Human Demonstrationcited by 0
  8. 2024Adaptive Querying for Reward Learning from Human Feedbackcited by 0
  9. 2025Supervised Reward Inferencecited by 0
  10. 2026Reward Learning through Ranking Mean Squared Errorcited by 0
  11. 2026MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inferencecited by 0
  12. 2026SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfercited by 0
  13. 2026Mitigating Cognitive Bias in RLHF by Altering Rationalitycited by 0
  14. 2026A Fair Objective for Human-Empowerment-Preserving AI: Desiderata, Design, and Likely Behavioral Consequencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization
  2. 2025Understanding Impact of Human Feedback via Influence Functions
  3. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
  4. 2023Active Reward Learning from Multiple Teachers
  5. 2025Capturing Individual Human Preferences with Reward Features
  6. 2023The History and Risks of Reinforcement Learning and Human Feedback
  7. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  8. 2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks