mlsanityData to 7 Oct 2026

Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proof

arXiv 2403.14593 ↗First posted 21 Mar 2024cs.LG stat.MLcited by 0

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Adversarial inverse reinforcement learning (AIRL) stands as a cornerstone approach in imitation learning, yet it faces criticisms from prior studies. In this paper, we rethink AIRL and respond to these criticisms. Criticism 1 lies in Inadequate Policy Imitation. We show that substituting the built-in algorithm with soft actor-critic (SAC) during policy updating (requires multi-iterations) significantly enhances the efficiency of policy imitation. Criticism 2 lies in Limited Performance in Transferable Reward Recovery Despite SAC Integration. While we find that SAC indeed exhibits a significant improvement in policy imitation, it introduces drawbacks to transferable reward recovery. We prove that the SAC algorithm itself is not feasible to disentangle the reward function comprehensively during the AIRL training process, and propose a hybrid framework, PPO-AIRL + SAC, for a satisfactory transfer effect. Criticism 3 lies in Unsatisfactory Proof from the Perspective of Potential Equilibrium. We reanalyze it from an algebraic theory perspective.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 45 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

29 of 45 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations28 Mar
  2. 2023Inverse Reinforcement Learning without Reinforcement Learning26 Mar
  3. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning22 Sep
  4. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching9 Sep
  5. 2021Identifiability in inverse reinforcement learning7 Jun
  6. 2020Demonstration-efficient Inverse Reinforcement Learning in Procedurally Generated Environments4 Dec
  7. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  8. 2020Off-Policy Adversarial Inverse Reinforcement Learning3 May
  9. 2018Inverse reinforcement learning for video games24 Oct
  10. 2018Adversarial Imitation via Variational Inverse Reinforcement Learning17 Sep

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016OpenAI Gymcited by 1,684
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  10. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  11. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  12. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
7 more
  1. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  2. 2019State Alignment-based Imitation Learningcited by 73
  3. 2021What Matters for Adversarial Imitation Learning?cited by 70
  4. 2018Multi-Agent Imitation Learning for Driving Simulationcited by 49
  5. 2020Energy-Based Imitation Learningcited by 39
  6. 2020State-only Imitation with Transition Dynamics Mismatchcited by 37
  7. 2023A Coupled Flow Approach to Imitation Learningcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks
  2. 2020oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions
  3. 2024On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theory
  4. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching
  5. 2019Multi-Agent Adversarial Inverse Reinforcement Learning
  6. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shaping
  7. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  8. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment