# Towards Resolving Unidentifiability in Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/1601.06569 · HTML: https://mlsanity.com/paper/1601.06569/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jan 2016.

## Built on by (most cited first, 18 of 18)

- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [Identifiability in inverse reinforcement learning](https://mlsanity.com/paper/2106.03498/) (2021)
- [General agents contain world models](https://mlsanity.com/paper/2506.01622/) (2025)
- [Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization](https://mlsanity.com/paper/2011.08541/) (2020)
- [Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning](https://mlsanity.com/paper/2209.10974/) (2022)
- [Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning](https://mlsanity.com/paper/2310.05422/) (2023)
- [Inverse Decision Modeling: Learning Interpretable Representations of Behavior](https://mlsanity.com/paper/2310.18591/) (2023)
- [Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations](https://mlsanity.com/paper/1907.03976/) (2019)
- [Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemble](https://mlsanity.com/paper/2206.00238/) (2022)
- [How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach](https://mlsanity.com/paper/2406.03812/) (2024)
- [Inverse Reinforcement Learning with Sub-optimal Experts](https://mlsanity.com/paper/2401.03857/) (2024)
- [The Limits of Predicting Agents from Behaviour](https://mlsanity.com/paper/2506.02923/) (2025)
- [Eliciting Risk Aversion with Inverse Reinforcement Learning via Interactive Questioning](https://mlsanity.com/paper/2308.08427/) (2023)
- [Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2211.15542/) (2022)
- [Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids](https://mlsanity.com/paper/2509.12010/) (2025)
- [Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL](https://mlsanity.com/paper/2510.06092/) (2025)
- [Inference of Utilities and Time Preference in Sequential Decision-Making](https://mlsanity.com/paper/2405.15975/) (2024)
- [Robustness in the Face of Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2501.06376/) (2025)

## Abstract (arXiv)

We consider a setting for Inverse Reinforcement Learning (IRL) where the learner is extended with the ability to actively select multiple environments, observing an agent's behavior on each environment. We first demonstrate that if the learner can experiment with any transition dynamics on some fixed set of states and actions, then there exists an algorithm that reconstructs the agent's reward function to the fullest extent theoretically possible, and that requires only a small (logarithmic) number of experiments. We contrast this result to what is known about IRL in single fixed environments, namely that the true reward function is fundamentally unidentifiable. We then extend this setting to the more realistic case where the learner may not select any transition dynamic, but rather is restricted to some fixed set of environments that it may try. We connect the problem of maximizing the information derived from experiments to submodular function maximization and demonstrate that a greedy algorithm is near optimal (up to logarithmic factors). Finally, we empirically validate our algorithm on an environment inspired by behavioral psychology.