# Partial Identifiability and Misspecification in Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2411.15951 · HTML: https://mlsanity.com/paper/2411.15951/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 10 of its 63 references that are in the map.
- First posted on arXiv 24 Nov 2024.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 10 of 10)

- [Categorizing Variants of Goodhart's Law](https://mlsanity.com/paper/1803.04585/) (2018)
- [What Matters for Adversarial Imitation Learning?](https://mlsanity.com/paper/2106.00672/) (2021)
- [Penalizing side effects using stepwise relative reachability](https://mlsanity.com/paper/1806.01186/) (2018)
- [Understanding Learned Reward Functions](https://mlsanity.com/paper/2012.05862/) (2020)
- [Avoiding Side Effects By Considering Future Tasks](https://mlsanity.com/paper/2010.07877/) (2020)
- [Reward Gaming in Conditional Text Generation](https://mlsanity.com/paper/2211.08714/) (2022)
- [Human irrationality: both bad and good for reward inference](https://mlsanity.com/paper/2111.06956/) (2021)
- [Dynamics-Aware Comparison of Learned Reward Functions](https://mlsanity.com/paper/2201.10081/) (2022)
- [Choice Set Misspecification in Reward Inference](https://mlsanity.com/paper/2101.07691/) (2021)
- [Preprocessing Reward Functions for Interpretability](https://mlsanity.com/paper/2203.13553/) (2022)

## Built on by (most cited first, 4 of 4)

- [Inverse Reinforcement Learning with Just Classification and a Few Regressions](https://mlsanity.com/paper/2509.21172/) (2025)
- [Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models](https://mlsanity.com/paper/2512.24407/) (2025)
- [Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL](https://mlsanity.com/paper/2510.06092/) (2025)
- [Causal Foundations of Collective Agency](https://mlsanity.com/paper/2605.00248/) (2026)

## Abstract (arXiv)

The aim of Inverse Reinforcement Learning (IRL) is to infer a reward function R from a policy π. This problem is difficult, for several reasons. First of all, there are typically multiple reward functions which are compatible with a given policy; this means that the reward function is only *partially identifiable*, and that IRL contains a certain fundamental degree of ambiguity. Secondly, in order to infer R from π, an IRL algorithm must have a *behavioural model* of how π relates to R. However, the true relationship between human preferences and human behaviour is very complex, and practically impossible to fully capture with a simple model. This means that the behavioural model in practice will be *misspecified*, which raises the worry that it might lead to unsound inferences if applied to real-world data. In this paper, we provide a comprehensive mathematical analysis of partial identifiability and misspecification in IRL. Specifically, we fully characterise and quantify the ambiguity of the reward function for all of the behavioural models that are most common in the current IRL literature. We also provide necessary and sufficient conditions that describe precisely how the observed demonstrator policy may differ from each of the standard behavioural models before that model leads to faulty inferences about the reward function R. In addition to this, we introduce a cohesive framework for reasoning about partial identifiability and misspecification in IRL, together with several formal tools that can be used to easily derive the partial identifiability and misspecification robustness of new IRL models, or analyse other kinds of reward learning algorithms.