mlsanityData to 7 Oct 2026

On the Sensitivity of Reward Inference to Misspecified Human Models

arXiv 2212.04717 ↗First posted 9 Dec 2022cs.LG cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Inferring reward functions from human behavior is at the center of value alignment - aligning AI objectives with what we, humans, actually want. But doing so relies on models of how humans behave given their objectives. After decades of research in cognitive science, neuroscience, and behavioral economics, obtaining accurate human models remains an open research topic. This begs the question: how accurate do these models need to be in order for the reward inference to be accurate? On the one hand, if small errors in the model can lead to catastrophic error in inference, the entire framework of reward learning seems ill-fated, as we will never have perfect models of human behavior. On the other hand, if as our models improve, we can have a guarantee that reward accuracy also improves, this would show the benefit of more work on the modeling side. We study this question both theoretically and empirically. We do show that it is unfortunately possible to construct small adversarial biases in behavior that lead to arbitrarily large errors in the inferred reward. However, and arguably more importantly, we are also able to identify reasonable assumptions under which the reward inference error can be bounded linearly in the error in the human model. Finally, we verify our theoretical insights in discrete and continuous control tasks with simulated and human data.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

16 of 33 references.

References · 16

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  3. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  4. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  5. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  6. 2018Shared Autonomy via Deep Reinforcement Learningcited by 65
  7. 2020Quantifying Differences in Reward Functionscited by 62
  8. 2021Identifiability in inverse reinforcement learningcited by 55
  9. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  10. 2015Shared Autonomy via Hindsight Optimizationcited by 46
  11. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  12. 2018Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behaviorcited by 40
4 more
  1. 2021Human irrationality: both bad and good for reward inferencecited by 25
  2. 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learningcited by 13
  3. 2017Risk-sensitive Inverse Reinforcement Learning via Semi- and Non-Parametric Methodscited by 12
  4. 2020Assisted Perception: Optimizing Observations to Communicate Statecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 12; 2025: 4; 2026: 2.

25 papers.

Sort:

By area:

  1. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  4. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  5. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  6. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  7. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  8. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  9. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  10. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  11. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  12. 2024Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generationcited by 9
13 more
  1. 2023Automatic Pair Construction for Contrastive Post-trainingcited by 7
  2. 2023Active teacher selection for reward learningcited by 7
  3. 2024HAF-RM: A Hybrid Alignment Framework for Reward Model Trainingcited by 5
  4. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecificationcited by 4
  5. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignmentcited by 3
  6. 2024VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedbackcited by 2
  7. 2024Solving the Inverse Alignment Problem for Efficient RLHFcited by 1
  8. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 1
  9. 2023PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Rewardcited by 0
  10. 2025Supervised Reward Inferencecited by 0
  11. 2025Bootstrapping LLMs via Preference-Based Policy Optimizationcited by 0
  12. 2026Mitigating Cognitive Bias in RLHF by Altering Rationalitycited by 0
  13. 2026Learning social norms enhances compatibility in dynamic human-AI coordinationcited by 0