mlsanityData to 7 Oct 2026

Identifiability and Generalizability in Constrained Inverse Reinforcement Learning

arXiv 2306.00629 ↗First posted 1 Jun 2023cs.LG cs.AI cs.SYcited by 16

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Two main challenges in Reinforcement Learning (RL) are designing appropriate reward functions and ensuring the safety of the learned policy. To address these challenges, we present a theoretical framework for Inverse Reinforcement Learning (IRL) in constrained Markov decision processes. From a convex-analytic perspective, we extend prior results on reward identifiability and generalizability to both the constrained setting and a more general class of regularizations. In particular, we show that identifiability up to potential shaping (Cao et al., 2021) is a consequence of entropy regularization and may generally no longer hold for other regularizations or in the presence of safety constraints. We also show that to ensure generalizability to new transition laws and constraints, the true reward must be identified up to a constant. Additionally, we derive a finite sample guarantee for the suboptimality of the learned rewards, and validate our results in a gridworld environment.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 63 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

31 of 63 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees4 Oct
  2. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning22 Sep
  3. 2022X-MEN: Guaranteed XOR-Maximum Entropy Constrained Inverse Reinforcement Learning22 Mar
  4. 2021Identifiability in inverse reinforcement learning7 Jun
  5. 2020Regularized Inverse Reinforcement Learning7 Oct
  6. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch2 Jul
  7. 2019Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints2 Jun
  8. 2018Maximum Causal Tsallis Entropy Imitation Learning22 May

Other references · 23

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2021Convex optimizationcited by 2,479
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2019A Theory of Regularized Markov Decision Processescited by 279
  9. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  10. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  11. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  12. 2017A unified view of entropy-regularized Markov decision processescited by 174
11 more
  1. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  2. 2018The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimizationcited by 98
  3. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  4. 2019Provably Efficient Imitation Learning from Observation Alonecited by 78
  5. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
  6. 2019Wasserstein Adversarial Imitation Learningcited by 59
  7. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  8. 2020Inverse Constrained Reinforcement Learningcited by 45
  9. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 36
  10. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 33
  11. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 5; 2026: 4.

16 papers.

Sort:

By area:

  1. 2023A safe exploration approach to constrained Markov decision processescited by 10
  2. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  3. 2024Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learningcited by 6
  4. 2026Safe Exploration via Policy Priorscited by 6
  5. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecificationcited by 4
  6. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  7. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3
  8. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithmcited by 2
  9. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroidscited by 1
  10. 2026Fast Rates for Inverse Reinforcement Learningcited by 1
  11. 2023A Novel Variational Lower Bound for Inverse Reinforcement Learningcited by 0
  12. 2024Inference of Utilities and Time Preference in Sequential Decision-Makingcited by 0
4 more
  1. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0
  2. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learningcited by 0
  3. 2026Multi-Objective Constraint Inference using Inverse reinforcement learningcited by 0
  4. 2026Priced Motion Through Optimal Faces: A Normal-Fan Geometry for Non-Stationary Adversarial MDPscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discounting
  2. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  3. 2021A Generalised Inverse Reinforcement Learning Framework
  4. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach
  5. 2023Inverse Reinforcement Learning With Constraint Recovery
  6. 2025Reward Compatibility: A Framework for Inverse RL
  7. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  8. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning