mlsanityData to 7 Oct 2026

Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning

arXiv 2209.10974 ↗First posted 22 Sep 2022cs.LGcited by 16

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

While Reinforcement Learning (RL) aims to train an agent from a reward function in a given environment, Inverse Reinforcement Learning (IRL) seeks to recover the reward function from observing an expert's behavior. It is well known that, in general, various reward functions can lead to the same optimal policy, and hence, IRL is ill-defined. However, (Cao et al., 2021) showed that, if we observe two or more experts with different discount factors or acting in different environments, the reward function can under certain conditions be identified up to a constant. This work starts by showing an equivalent identifiability statement from multiple experts in tabular MDPs based on a rank condition, which is easily verifiable and is shown to be also necessary. We then extend our result to various different scenarios, i.e., we characterize reward identifiability in the case where the reward function can be represented as a linear combination of given features, making it more interpretable, or when we have access to approximate transition matrices. Even when the reward is not identifiable, we provide conditions characterizing when data on multiple experts in a given environment allows to generalize and train an optimal agent in a new environment. Our theoretical results on reward identifiability and generalizability are validated in various numerical experiments.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 94 references that are in the map. First posted on arXiv 22 Sep 2022.

Builds on

34 of 94 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021Identifiability in inverse reinforcement learning7 Jun
  2. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization17 Nov
  3. 2020Inverse Reinforcement Learning from a Gradient-based Learner15 Jul
  4. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch2 Jul
  5. 2019Option Compatible Reward Inverse Reinforcement Learning7 Nov
  6. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations12 Apr

Other references · 28

  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  3. 2017Robust Adversarial Reinforcement Learningcited by 439
  4. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  5. 2019A Theory of Regularized Markov Decision Processescited by 279
  6. 2018Generative Adversarial Imitation from Observationcited by 183
  7. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  8. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  9. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  10. 2021On the Expressivity of Markov Rewardcited by 71
  11. 2020Quantifying Differences in Reward Functionscited by 62
  12. 2020Reinforcement Learning in Economics and Financecited by 57
16 more
  1. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
  2. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  3. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  4. 2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
  5. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 44
  6. 2020State-only Imitation with Transition Dynamics Mismatchcited by 37
  7. 2017Repeated Inverse Reinforcement Learningcited by 29
  8. 2018Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learningcited by 24
  9. 2020Value Alignment Verificationcited by 20
  10. 2016Towards Resolving Unidentifiability in Inverse Reinforcement Learningcited by 18
  11. 2016Avoiding Wireheading with Value Reinforcement Learningcited by 15
  12. 2022Robust Learning from Observation with Model Misspecificationcited by 9
  13. 2017Factoring Exogenous State for Model-Free Monte Carlocited by 3
  14. 2019New developments in revealed preference theory: decisions under risk, uncertainty, and intertemporal choicecited by 2
  15. 2021Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learningcited by 2
  16. 2013Solving Relational MDPs with Exogenous Events and Additive Rewardscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 7; 2025: 4; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  2. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learningcited by 16
  3. 2024Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learningcited by 6
  4. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  5. 2022Environment Design for Inverse Reinforcement Learningcited by 4
  6. 2024Inverse Reinforcement Learning with Sub-optimal Expertscited by 4
  7. 2024On Feasible Rewards in Multi-Agent Inverse Reinforcement Learningcited by 4
  8. 2025Inversely Learning Transferable Rewards via Abstracted Statescited by 1
  9. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroidscited by 1
  10. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proofcited by 0
  11. 2024Randomized algorithms and PAC bounds for inverse reinforcement learning in continuous spacescited by 0
  12. 2024Inverse Reinforcement Learning with Multiple Planning Horizonscited by 0
4 more
  1. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0
  2. 2025Blind Inverse Game Theory: Jointly Decoding Rewards and Rationality in Entropy-Regularized Competitive Gamescited by 0
  3. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learningcited by 0
  4. 2026Inverse Reinforcement Learning for Interpretable Keystroke Biomarkers in Parkinson's Diseasecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Partial Identifiability and Misspecification in Inverse Reinforcement Learning
  2. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms
  3. 2024Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discounting
  4. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  5. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
  6. 2025Reward Compatibility: A Framework for Inverse RL
  7. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles
  8. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress