# Identifiability and Generalizability in Constrained Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2306.00629 · HTML: https://mlsanity.com/paper/2306.00629/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 31 of its 63 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 30 of 31)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [A Theory of Regularized Markov Decision Processes](https://mlsanity.com/paper/1901.11275/) (2019)
- [On the Global Convergence Rates of Softmax Policy Gradient Methods](https://mlsanity.com/paper/2005.06392/) (2020)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [A unified view of entropy-regularized Markov decision processes](https://mlsanity.com/paper/1705.07798/) (2017)
- [Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization](https://mlsanity.com/paper/2007.06558/) (2020)
- [The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimization](https://mlsanity.com/paper/1807.03907/) (2018)
- [Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap](https://mlsanity.com/paper/2103.03236/) (2021)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [Safe Reinforcement Learning via Curriculum Induction](https://mlsanity.com/paper/2006.12136/) (2020)
- [Wasserstein Adversarial Imitation Learning](https://mlsanity.com/paper/1906.08113/) (2019)
- [Identifiability in inverse reinforcement learning](https://mlsanity.com/paper/2106.03498/) (2021)
- [Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2011.09999/) (2020)
- [Invariance in Policy Optimisation and Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2203.07475/) (2022)
- [Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees](https://mlsanity.com/paper/2210.01808/) (2022)
- [Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learning](https://mlsanity.com/paper/1709.06293/) (2017)
- [A Dual Approach to Constrained Markov Decision Processes with Entropy Regularization](https://mlsanity.com/paper/2110.08923/) (2021)
- [Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch](https://mlsanity.com/paper/2007.01174/) (2020)
- [Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs](https://mlsanity.com/paper/2206.02346/) (2022)
- [Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints](https://mlsanity.com/paper/1906.00429/) (2019)
- [Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning](https://mlsanity.com/paper/2209.10974/) (2022)
- [Maximum Causal Tsallis Entropy Imitation Learning](https://mlsanity.com/paper/1805.08336/) (2018)
- [Regularized Inverse Reinforcement Learning](https://mlsanity.com/paper/2010.03691/) (2020)

## Built on by (most cited first, 16 of 16)

- [A safe exploration approach to constrained Markov decision processes](https://mlsanity.com/paper/2312.00561/) (2023)
- [A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges](https://mlsanity.com/paper/2409.07569/) (2024)
- [Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning](https://mlsanity.com/paper/2406.01793/) (2024)
- [Safe Exploration via Policy Priors](https://mlsanity.com/paper/2601.19612/) (2026)
- [Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification](https://mlsanity.com/paper/2403.06854/) (2024)
- [MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrations](https://mlsanity.com/paper/2505.18595/) (2025)
- [Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design](https://mlsanity.com/paper/2506.09508/) (2025)
- [Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm](https://mlsanity.com/paper/2403.16829/) (2024)
- [Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids](https://mlsanity.com/paper/2509.12010/) (2025)
- [Fast Rates for Inverse Reinforcement Learning](https://mlsanity.com/paper/2605.14599/) (2026)
- [A Novel Variational Lower Bound for Inverse Reinforcement Learning](https://mlsanity.com/paper/2311.03698/) (2023)
- [Inference of Utilities and Time Preference in Sequential Decision-Making](https://mlsanity.com/paper/2405.15975/) (2024)
- [Robustness in the Face of Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2501.06376/) (2025)
- [Statistical analysis of Inverse Entropy-regularized Reinforcement Learning](https://mlsanity.com/paper/2512.06956/) (2025)
- [Multi-Objective Constraint Inference using Inverse reinforcement learning](https://mlsanity.com/paper/2605.06951/) (2026)
- [Priced Motion Through Optimal Faces: A Normal-Fan Geometry for Non-Stationary Adversarial MDPs](https://mlsanity.com/paper/2606.29092/) (2026)

## Abstract (arXiv)

Two main challenges in Reinforcement Learning (RL) are designing appropriate reward functions and ensuring the safety of the learned policy. To address these challenges, we present a theoretical framework for Inverse Reinforcement Learning (IRL) in constrained Markov decision processes. From a convex-analytic perspective, we extend prior results on reward identifiability and generalizability to both the constrained setting and a more general class of regularizations. In particular, we show that identifiability up to potential shaping (Cao et al., 2021) is a consequence of entropy regularization and may generally no longer hold for other regularizations or in the presence of safety constraints. We also show that to ensure generalizability to new transition laws and constraints, the true reward must be identified up to a constant. Additionally, we derive a finite sample guarantee for the suboptimality of the learned rewards, and validate our results in a gridworld environment.