mlsanityData to 7 Oct 2026

On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theory

arXiv 2410.07643 ↗First posted 10 Oct 2024stat.ML cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

In the context of inverse reinforcement learning (IRL) with a single expert, adversarial inverse reinforcement learning (AIRL) serves as a foundational approach to providing comprehensive and transferable task descriptions. However, AIRL faces practical performance challenges, primarily stemming from the framework's overly idealized decomposability condition, the unclear proof regarding the potential equilibrium in reward recovery, or questionable robustness in high-dimensional environments. This paper revisits AIRL in high-dimensional scenarios where the state space tends to infinity. Specifically, we first establish a necessary and sufficient condition for reward transferability by examining the rank of the matrix derived from subtracting the identity matrix from the transition matrix. Furthermore, leveraging random matrix theory, we analyze the spectral distribution of this matrix, demonstrating that our rank criterion holds with high probability even when the transition matrices are unobservable. This suggests that the limitations on transfer are not inherent to the AIRL framework itself, but are instead related to the training variance of the reinforcement learning algorithms employed within it. Based on this insight, we propose a hybrid framework that integrates on-policy proximal policy optimization in the source environment with off-policy soft actor-critic in the target environment, leading to significant improvements in reward transfer effectiveness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 43 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

22 of 43 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations28 Mar
  2. 2022Receding Horizon Inverse Reinforcement Learning9 Jun
  3. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching9 Sep
  4. 2021Identifiability in inverse reinforcement learning7 Jun
  5. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  6. 2020Off-Policy Adversarial Inverse Reinforcement Learning3 May
  7. 2018Adversarial Imitation via Variational Inverse Reinforcement Learning17 Sep

Other references · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016OpenAI Gymcited by 1,684
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  7. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  8. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  9. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  10. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  11. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  12. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
3 more
  1. 2019State Alignment-based Imitation Learningcited by 73
  2. 2018Multi-Agent Imitation Learning for Driving Simulationcited by 49
  3. 2020Energy-Based Imitation Learningcited by 39

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proof
  2. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shaping
  3. 2020oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions
  4. 2019Multi-Agent Adversarial Inverse Reinforcement Learning
  5. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  6. 2025Hybrid-AIRL: Enhancing Inverse Reinforcement Learning with Supervised Expert Guidance
  7. 2024Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning
  8. 2019Adversarial recovery of agent rewards from latent spaces of the limit order book