mlsanityData to 7 Oct 2026

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning

arXiv 2406.01793 ↗First posted 3 Jun 2024cs.LG cs.AI stat.MLcited by 6

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse reinforcement learning (IRL) aims to infer a reward from expert demonstrations, motivated by the idea that the reward, rather than the policy, is the most succinct and transferable description of a task [Ng et al., 2000]. However, the reward corresponding to an optimal policy is not unique, making it unclear if an IRL-learned reward is transferable to new transition laws in the sense that its optimal policy aligns with the optimal policy corresponding to the expert's true reward. Past work has addressed this problem only under the assumption of full access to the expert's policy, guaranteeing transferability when learning from two experts with the same reward but different transition laws that satisfy a specific rank condition [Rolland et al., 2022]. In this work, we show that the conditions developed under full access to the expert's policy cannot guarantee transferability in the more practical scenario where we have access only to demonstrations of the expert. Instead of a binary rank condition, we propose principal angles as a more refined measure of similarity and dissimilarity between transition laws. Based on this, we then establish two key results: 1) a sufficient condition for transferability to any transition laws when learning from at least two experts with sufficiently different transition laws, and 2) a sufficient condition for transferability to local changes in the transition law when learning from a single expert. Furthermore, we also provide a probably approximately correct (PAC) algorithm and an end-to-end analysis for learning transferable rewards from demonstrations of multiple experts.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 47 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

21 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learning1 Jun
  2. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning22 Sep
  3. 2021Identifiability in inverse reinforcement learning7 Jun
  4. 2020Regularized Inverse Reinforcement Learning7 Oct

Other references · 17

  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2021Convex optimizationcited by 2,479
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  5. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  6. 2019A Theory of Regularized Markov Decision Processescited by 279
  7. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  8. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  9. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  10. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  11. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 124
  12. 2018Tsallis-INF: An Optimal Algorithm for Stochastic and Adversarial Banditscited by 123
5 more
  1. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  2. 2017Repeated Inverse Reinforcement Learningcited by 29
  3. 2015Information-Theoretic Bounded Rationalitycited by 21
  4. 2023Best of Both Worlds Policy Optimizationcited by 17
  5. 2023Learning in Zero-Sum Markov Games: Relaxing Strong Reachability and Mixing Time Assumptionscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Designcited by 3
  2. 2025Inversely Learning Transferable Rewards via Abstracted Statescited by 1
  3. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroidscited by 1
  4. 2026Fast Rates for Inverse Reinforcement Learningcited by 1
  5. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0
  6. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  2. 2025Reward Compatibility: A Framework for Inverse RL
  3. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles
  4. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  5. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  6. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
  7. 2019Option Compatible Reward Inverse Reinforcement Learning
  8. 2023Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations