mlsanityData to 7 Oct 2026

Off-Policy Adversarial Inverse Reinforcement Learning

arXiv 2005.01138 ↗First posted 3 May 2020cs.LG cs.AI cs.ROcited by 8

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Adversarial Imitation Learning (AIL) is a class of algorithms in Reinforcement learning (RL), which tries to imitate an expert without taking any reward from the environment and does not provide expert behavior directly to the policy training. Rather, an agent learns a policy distribution that minimizes the difference from expert behavior in an adversarial setting. Adversarial Inverse Reinforcement Learning (AIRL) leverages the idea of AIL, integrates a reward function approximation along with learning the policy, and shows the utility of IRL in the transfer learning setting. But the reward function approximator that enables transfer learning does not perform well in imitation tasks. We propose an Off-Policy Adversarial Inverse Reinforcement Learning (Off-policy-AIRL) algorithm which is sample efficient as well as gives good imitation performance compared to the state-of-the-art AIL algorithm in the continuous control tasks. For the same reward function approximator, we show the utility of learning our algorithm over AIL by using the learned reward function to retrain the policy over a task under significant variation where expert demonstrations are absent.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 12 of its 25 references that are in the map. First posted on arXiv 3 May 2020.

Builds on

12 of 25 references.

References · 12

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2017Improved Training of Wasserstein GANscited by 2,223
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  9. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  10. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  11. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 103
  12. 2018Reinforcement Learning for Improving Agent Designcited by 55

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 1; 2024: 3.

8 papers.

  1. 2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matchingcited by 15
  2. 2021Importance of Empirical Sample Complexity Analysis for Offline Reinforcement Learningcited by 3
  3. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3
  4. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
  5. 2022Weighted Maximum Entropy Inverse Reinforcement Learningcited by 0
  6. 2022Classifying Ambiguous Identities in Hidden-Role Stochastic Games with Multi-Agent Reinforcement Learningcited by 0
  7. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proofcited by 0
  8. 2024On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks
  2. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching
  3. 2025Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization
  4. 2019Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning
  5. 2024Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guarantees
  6. 2020oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions
  7. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms
  8. 2023Adversarial Imitation Learning On Aggregated Data