# Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2209.10974 · HTML: https://mlsanity.com/paper/2209.10974/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 34 of its 94 references that are in the map.
- First posted on arXiv 22 Sep 2022.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 30 of 34)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [A Theory of Regularized Markov Decision Processes](https://mlsanity.com/paper/1901.11275/) (2019)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Action Robust Reinforcement Learning and Applications in Continuous Control](https://mlsanity.com/paper/1901.09184/) (2019)
- [Reward (Mis)design for Autonomous Driving](https://mlsanity.com/paper/2104.13906/) (2021)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [On the Expressivity of Markov Reward](https://mlsanity.com/paper/2111.00876/) (2021)
- [Quantifying Differences in Reward Functions](https://mlsanity.com/paper/2006.13900/) (2020)
- [Reinforcement Learning in Economics and Finance](https://mlsanity.com/paper/2003.10014/) (2020)
- [Identifiability in inverse reinforcement learning](https://mlsanity.com/paper/2106.03498/) (2021)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Model-Free Imitation Learning with Policy Optimization](https://mlsanity.com/paper/1605.08478/) (2016)
- [Invariance in Policy Optimisation and Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2203.07475/) (2022)
- [Occam's razor is insufficient to infer the preferences of irrational agents](https://mlsanity.com/paper/1712.05812/) (2017)
- [Robust Reinforcement Learning via Adversarial training with Langevin Dynamics](https://mlsanity.com/paper/2002.06063/) (2020)
- [State-only Imitation with Transition Dynamics Mismatch](https://mlsanity.com/paper/2002.11879/) (2020)
- [Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch](https://mlsanity.com/paper/2007.01174/) (2020)
- [Repeated Inverse Reinforcement Learning](https://mlsanity.com/paper/1705.05427/) (2017)
- [Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learning](https://mlsanity.com/paper/1806.01584/) (2018)
- [Value Alignment Verification](https://mlsanity.com/paper/2012.01557/) (2020)
- [Towards Resolving Unidentifiability in Inverse Reinforcement Learning](https://mlsanity.com/paper/1601.06569/) (2016)
- [Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization](https://mlsanity.com/paper/2011.08541/) (2020)
- [Avoiding Wireheading with Value Reinforcement Learning](https://mlsanity.com/paper/1605.03143/) (2016)
- [Robust Learning from Observation with Model Misspecification](https://mlsanity.com/paper/2202.06003/) (2022)
- [Inverse Reinforcement Learning from a Gradient-based Learner](https://mlsanity.com/paper/2007.07812/) (2020)
- [Factoring Exogenous State for Model-Free Monte Carlo](https://mlsanity.com/paper/1703.09390/) (2017)

## Built on by (most cited first, 16 of 16)

- [When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.07457/) (2023)
- [Identifiability and Generalizability in Constrained Inverse Reinforcement Learning](https://mlsanity.com/paper/2306.00629/) (2023)
- [Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning](https://mlsanity.com/paper/2406.01793/) (2024)
- [Grounded Answers for Multi-agent Decision-making Problem through Generative World Model](https://mlsanity.com/paper/2410.02664/) (2024)
- [On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning](https://mlsanity.com/paper/2411.15046/) (2024)
- [Environment Design for Inverse Reinforcement Learning](https://mlsanity.com/paper/2210.14972/) (2022)
- [Inverse Reinforcement Learning with Sub-optimal Experts](https://mlsanity.com/paper/2401.03857/) (2024)
- [Inversely Learning Transferable Rewards via Abstracted States](https://mlsanity.com/paper/2501.01669/) (2025)
- [Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids](https://mlsanity.com/paper/2509.12010/) (2025)
- [Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proof](https://mlsanity.com/paper/2403.14593/) (2024)
- [Randomized algorithms and PAC bounds for inverse reinforcement learning in continuous spaces](https://mlsanity.com/paper/2405.15509/) (2024)
- [Inverse Reinforcement Learning with Multiple Planning Horizons](https://mlsanity.com/paper/2409.18051/) (2024)
- [Robustness in the Face of Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2501.06376/) (2025)
- [Blind Inverse Game Theory: Jointly Decoding Rewards and Rationality in Entropy-Regularized Competitive Games](https://mlsanity.com/paper/2511.05640/) (2025)
- [Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning](https://mlsanity.com/paper/2606.22056/) (2026)
- [Inverse Reinforcement Learning for Interpretable Keystroke Biomarkers in Parkinson's Disease](https://mlsanity.com/paper/2606.25270/) (2026)

## Abstract (arXiv)

While Reinforcement Learning (RL) aims to train an agent from a reward function in a given environment, Inverse Reinforcement Learning (IRL) seeks to recover the reward function from observing an expert's behavior. It is well known that, in general, various reward functions can lead to the same optimal policy, and hence, IRL is ill-defined. However, (Cao et al., 2021) showed that, if we observe two or more experts with different discount factors or acting in different environments, the reward function can under certain conditions be identified up to a constant. This work starts by showing an equivalent identifiability statement from multiple experts in tabular MDPs based on a rank condition, which is easily verifiable and is shown to be also necessary. We then extend our result to various different scenarios, i.e., we characterize reward identifiability in the case where the reward function can be represented as a linear combination of given features, making it more interpretable, or when we have access to approximate transition matrices. Even when the reward is not identifiable, we provide conditions characterizing when data on multiple experts in a given environment allows to generalize and train an optimal agent in a new environment. Our theoretical results on reward identifiability and generalizability are validated in various numerical experiments.