mlsanityData to 7 Oct 2026

Partial Identifiability and Misspecification in Inverse Reinforcement Learning

arXiv 2411.15951 ↗First posted 24 Nov 2024cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function R from a policy π. This problem is difficult, for several reasons. First of all, there are typically multiple reward functions which are compatible with a given policy; this means that the reward function is only *partially identifiable*, and that IRL contains a certain fundamental degree of ambiguity. Secondly, in order to infer R from π, an IRL algorithm must have a *behavioural model* of how π relates to R. However, the true relationship between human preferences and human behaviour is very complex, and practically impossible to fully capture with a simple model. This means that the behavioural model in practice will be *misspecified*, which raises the worry that it might lead to unsound inferences if applied to real-world data. In this paper, we provide a comprehensive mathematical analysis of partial identifiability and misspecification in IRL. Specifically, we fully characterise and quantify the ambiguity of the reward function for all of the behavioural models that are most common in the current IRL literature. We also provide necessary and sufficient conditions that describe precisely how the observed demonstrator policy may differ from each of the standard behavioural models before that model leads to faulty inferences about the reward function R. In addition to this, we introduce a cohesive framework for reasoning about partial identifiability and misspecification in IRL, together with several formal tools that can be used to easily derive the partial identifiability and misspecification robustness of new IRL models, or analyse other kinds of reward learning algorithms.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 10 of its 63 references that are in the map. First posted on arXiv 24 Nov 2024.

Builds on

10 of 63 references.

References · 10

  1. 2018Categorizing Variants of Goodhart's Lawcited by 132
  2. 2021What Matters for Adversarial Imitation Learning?cited by 70
  3. 2018Penalizing side effects using stepwise relative reachabilitycited by 38
  4. 2020Understanding Learned Reward Functionscited by 37
  5. 2020Avoiding Side Effects By Considering Future Taskscited by 32
  6. 2022Reward Gaming in Conditional Text Generationcited by 29
  7. 2021Human irrationality: both bad and good for reward inferencecited by 25
  8. 2022Dynamics-Aware Comparison of Learned Reward Functionscited by 17
  9. 2021Choice Set Misspecification in Reward Inferencecited by 13
  10. 2022Preprocessing Reward Functions for Interpretabilitycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Inverse Reinforcement Learning with Just Classification and a Few Regressionscited by 4
  2. 2025Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Modelscited by 2
  3. 2025Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RLcited by 1
  4. 2026Causal Foundations of Collective Agencycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification
  2. 2022Misspecification in Inverse Reinforcement Learning
  3. 2024Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discounting
  4. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  6. 2026Quantifying Potential Observation Missingness in Inverse Reinforcement Learning
  7. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
  8. 2020Accounting for Human Learning when Inferring Human Preferences