mlsanityData to 7 Oct 2026

Misspecification in Inverse Reinforcement Learning

arXiv 2212.03201 ↗First posted 6 Dec 2022cs.LGcited by 27

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function R from a policy π. To do this, we need a model of how π relates to R. In the current literature, the most common models are optimality, Boltzmann rationality, and causal entropy maximisation. One of the primary motivations behind IRL is to infer human preferences from human behaviour. However, the true relationship between human preferences and human behaviour is much more complex than any of the models currently used in IRL. This means that they are misspecified, which raises the worry that they might lead to unsound inferences if applied to real-world data. In this paper, we provide a mathematical analysis of how robust different IRL models are to misspecification, and answer precisely how the demonstrator policy may differ from each of the standard models before that model leads to faulty inferences about the reward function R. We also introduce a framework for reasoning about misspecification in IRL, together with formal tools that can be used to easily derive the misspecification robustness of new IRL models.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 16 of its 42 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

16 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Identifiability in inverse reinforcement learning7 Jun

Other references · 15

  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  2. 2016Cooperative Inverse Reinforcement Learningcited by 298
  3. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  4. 2022Defining and Characterizing Reward Hackingcited by 184
  5. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  6. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  7. 2021What Matters for Adversarial Imitation Learning?cited by 70
  8. 2020Quantifying Differences in Reward Functionscited by 62
  9. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  10. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  11. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  12. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 44
3 more
  1. 2021Human irrationality: both bad and good for reward inferencecited by 25
  2. 2021Choice Set Misspecification in Reward Inferencecited by 13
  3. 2022Calculus on MDPs: Potential Shaping as a Gradientcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 10; 2025: 7; 2026: 1.

27 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023Goodhart's Law in Reinforcement Learningcited by 37
  3. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 29
  4. 2024When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedbackcited by 17
  5. 2025Jackpot! Alignment as a Maximal Lotterycited by 17
  6. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  7. 2023Active Reward Learning from Multiple Teacherscited by 13
  8. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Taskscited by 9
  9. 2023Active teacher selection for reward learningcited by 7
  10. 2023(Ir)rationality in AI: State of the Art, Research Challenges and Open Questionscited by 5
  11. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approachcited by 5
  12. 2025Emergent Risk Awareness in Rational Agents under Resource Constraintscited by 5
15 more
  1. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  2. 2025Learning Policy Committees for Effective Personalization in MDPs with Diverse Taskscited by 4
  3. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignmentcited by 3
  4. 2025The Limits of Predicting Agents from Behaviourcited by 3
  5. 2024Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learningcited by 2
  6. 2024Explaining Learned Reward Functions with Counterfactual Trajectoriescited by 1
  7. 2024Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatchcited by 1
  8. 2024Preserving the Privacy of Reward Functions in MDPs through Deceptioncited by 1
  9. 2024Provably Efficient Exploration in Inverse Constrained Reinforcement Learningcited by 1
  10. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroidscited by 1
  11. 2025On Discovering Algorithms for Adversarial Imitation Learningcited by 1
  12. 2023PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Rewardcited by 0
  13. 2023FP-IRL: Fokker--Planck Inverse Reinforcement Learning -- A Physics-Constrained Approach to Markov Decision Processescited by 0
  14. 2025Supervised Reward Inferencecited by 0
  15. 2026Understanding Goal Generalisation in Sequential Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification
  2. 2024Partial Identifiability and Misspecification in Inverse Reinforcement Learning
  3. 2020Accounting for Human Learning when Inferring Human Preferences
  4. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  5. 2025Efficient Imitation under Misspecification
  6. 2026Quantifying Potential Observation Missingness in Inverse Reinforcement Learning
  7. 2019Modeling and Interpreting Real-world Human Risk Decision Making with Inverse Reinforcement Learning
  8. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles