# Identifiability in inverse reinforcement learning

arXiv: https://arxiv.org/abs/2106.03498 · HTML: https://mlsanity.com/paper/2106.03498/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 8 of its 44 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 8 of 8)

- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Repeated Inverse Reinforcement Learning](https://mlsanity.com/paper/1705.05427/) (2017)
- [Towards Resolving Unidentifiability in Inverse Reinforcement Learning](https://mlsanity.com/paper/1601.06569/) (2016)
- [Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization](https://mlsanity.com/paper/2011.08541/) (2020)

## Built on by (most cited first, 30 of 55)

- [From r to Q^*: Your Language Model is Secretly a Q-Function](https://mlsanity.com/paper/2404.12358/) (2024)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Invariance in Policy Optimisation and Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2203.07475/) (2022)
- [Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees](https://mlsanity.com/paper/2210.01808/) (2022)
- [Misspecification in Inverse Reinforcement Learning](https://mlsanity.com/paper/2212.03201/) (2022)
- [When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.07457/) (2023)
- [On the Sensitivity of Reward Inference to Misspecified Human Models](https://mlsanity.com/paper/2212.04717/) (2022)
- [Coherent Soft Imitation Learning](https://mlsanity.com/paper/2305.16498/) (2023)
- [Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning](https://mlsanity.com/paper/2209.10974/) (2022)
- [Identifiability and Generalizability in Constrained Inverse Reinforcement Learning](https://mlsanity.com/paper/2306.00629/) (2023)
- [A Primer on Maximum Causal Entropy Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.11409/) (2022)
- [Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning](https://mlsanity.com/paper/2212.01174/) (2022)
- [K-SHAP: Policy Clustering Algorithm for Anonymous Multi-Agent State-Action Pairs](https://mlsanity.com/paper/2302.11996/) (2023)
- [Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning](https://mlsanity.com/paper/2406.01793/) (2024)
- [Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemble](https://mlsanity.com/paper/2206.00238/) (2022)
- [How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach](https://mlsanity.com/paper/2406.03812/) (2024)
- [Agency Is Frame-Dependent](https://mlsanity.com/paper/2502.04403/) (2025)
- [Inverse Reinforcement Learning with Just Classification and a Few Regressions](https://mlsanity.com/paper/2509.21172/) (2025)
- [Calculus on MDPs: Potential Shaping as a Gradient](https://mlsanity.com/paper/2208.09570/) (2022)
- [Environment Design for Inverse Reinforcement Learning](https://mlsanity.com/paper/2210.14972/) (2022)
- [FoMo Rewards: Can we cast foundation models as reward functions?](https://mlsanity.com/paper/2312.03881/) (2023)
- [Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification](https://mlsanity.com/paper/2403.06854/) (2024)
- [Boosting Soft Q-Learning by Bounding](https://mlsanity.com/paper/2406.18033/) (2024)
- [On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning](https://mlsanity.com/paper/2411.15046/) (2024)
- [Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment](https://mlsanity.com/paper/2410.23680/) (2024)
- [Bootstrapped Reward Shaping](https://mlsanity.com/paper/2501.00989/) (2025)
- [Learning Reward Machines from Partially Observed Policies](https://mlsanity.com/paper/2502.03762/) (2025)
- [Eliciting Risk Aversion with Inverse Reinforcement Learning via Interactive Questioning](https://mlsanity.com/paper/2308.08427/) (2023)
- [Toward Computationally Efficient Inverse Reinforcement Learning via Reward Shaping](https://mlsanity.com/paper/2312.09983/) (2023)
- [Inversely Learning Transferable Rewards via Abstracted States](https://mlsanity.com/paper/2501.01669/) (2025)

## Abstract (arXiv)

Inverse reinforcement learning attempts to reconstruct the reward function in a Markov decision problem, using observations of agent actions. As already observed in Russell [1998] the problem is ill-posed, and the reward function is not identifiable, even under the presence of perfect information about optimal behavior. We provide a resolution to this non-identifiability for problems with entropy regularization. For a given environment, we fully characterize the reward functions leading to a given policy and demonstrate that, given demonstrations of actions for the same reward under two distinct discount factors, or under sufficiently different environments, the unobserved reward can be recovered up to a constant. We also give general necessary and sufficient conditions for reconstruction of time-homogeneous rewards on finite horizons, and for action-independent rewards, generalizing recent results of Kim et al. [2021] and Fu et al. [2018].