mlsanityData to 7 Oct 2026

Choice Set Misspecification in Reward Inference

arXiv 2101.07691 ↗First posted 19 Jan 2021cs.AI cs.HCcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Specifying reward functions for robots that operate in environments without a natural reward signal can be challenging, and incorrectly specified rewards can incentivise degenerate or dangerous behavior. A promising alternative to manually specifying reward functions is to enable robots to infer them from human feedback, like demonstrations or corrections. To interpret this feedback, robots treat as approximately optimal a choice the person makes from a choice set, like the set of possible trajectories they could have demonstrated or possible corrections they could have made. In this work, we introduce the idea that the choice set itself might be difficult to specify, and analyze choice set misspecification: what happens as the robot makes incorrect assumptions about the set of choices from which the human selects their feedback. We propose a classification of different kinds of choice set misspecification, and show that these different classes lead to meaningful differences in the inferred reward and resulting performance. While we would normally expect misspecification to hurt, we find that certain kinds of misspecification are neither helpful nor harmful (in expectation). However, in other situations, misspecification can be extremely harmful, leading the robot to believe the opposite of what it should believe. We hope our results will allow for better prediction and response to the effects of misspecification in real-world reward inference.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 14 of its 27 references that are in the map. First posted on arXiv 19 Jan 2021.

Builds on

14 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 12

  1. 2016Cooperative Inverse Reinforcement Learningcited by 298
  2. 2017Inverse Reward Designcited by 246
  3. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  4. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  5. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  6. 2018Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behaviorcited by 40
  7. 2019Preferences Implicit in the State of the Worldcited by 33
  8. 2013Computational Rationalization: The Inverse Equilibrium Problemcited by 24
  9. 2020Quantifying Hypothesis Space Misspecification in Learning from Human-Robot Demonstrations and Physical Correctionscited by 19
  10. 2020LESS is More: Rethinking Probabilistic Models of Human Behaviorcited by 17
  11. 2019Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learningcited by 15
  12. 2018Active Inverse Reward Designcited by 12

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 5; 2024: 4; 2025: 1.

13 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  3. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  4. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  5. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  6. 2023Active Reward Learning from Multiple Teacherscited by 13
  7. 2021Uncertain Decisions Facilitate Better Preference Learningcited by 10
  8. 2023Active teacher selection for reward learningcited by 7
  9. 2023Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarizationcited by 5
  10. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  11. 2025Emergent Risk Awareness in Rational Agents under Resource Constraintscited by 5
  12. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecificationcited by 4
1 more
  1. 2024Partial Identifiability and Misspecification in Inverse Reinforcement Learningcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization
  2. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference
  3. 2023On The Fragility of Learned Reward Functions
  4. 2024Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatch
  5. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning
  6. 2025Efficient Imitation under Misspecification
  7. 2023REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback
  8. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking