mlsanityData to 7 Oct 2026

OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching

arXiv 2109.04307 ↗First posted 9 Sep 2021cs.LG cs.AI cs.ROcited by 15

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Inverse Reinforcement Learning (IRL) is attractive in scenarios where reward engineering can be tedious. However, prior IRL algorithms use on-policy transitions, which require intensive sampling from the current policy for stable and optimal performance. This limits IRL applications in the real world, where environment interactions can become highly expensive. To tackle this problem, we present Off-Policy Inverse Reinforcement Learning (OPIRL), which (1) adopts off-policy data distribution instead of on-policy and enables significant reduction of the number of interactions with the environment, (2) learns a stationary reward function that is transferable with high generalization capabilities on changing dynamics, and (3) leverages mode-covering behavior for faster convergence. We demonstrate that our method is considerably more sample efficient and generalizes to novel environments through the experiments. Our method achieves better or comparable results on policy performance baselines with significantly fewer interactions. Furthermore, we empirically show that the recovered reward function generalizes to different tasks where prior arts are prone to fail.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 48 references that are in the map. First posted on arXiv 9 Sep 2021.

Builds on

35 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  2. 2020Off-Policy Adversarial Inverse Reinforcement Learning3 May
  3. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variables20 Sep
  4. 2019Multi-Agent Adversarial Inverse Reinforcement Learning30 Jul
  5. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learning31 May

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2017Improved Training of Wasserstein GANscited by 2,223
  6. 2016OpenAI Gymcited by 1,684
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2016Generative Adversarial Imitation Learningcited by 1,576
  10. 2016End to End Learning for Self-Driving Carscited by 1,000
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
18 more
  1. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  2. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  3. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  4. 2016Cooperative Inverse Reinforcement Learningcited by 298
  5. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  6. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  8. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  9. 2021Off-Policy Imitation Learning from Observationscited by 70
  10. 2021What Matters for Adversarial Imitation Learning?cited by 70
  11. 2018Sample-Efficient Imitation Learning via Generative Adversarial Netscited by 36
  12. 2020Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Drivingcited by 34
  13. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  14. 2016Improving Policy Gradient by Exploring Under-appreciated Rewardscited by 14
  15. 2020Batch Exploration with Examples for Scalable Robotic Reinforcement Learningcited by 14
  16. 2021SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matchingcited by 14
  17. 2020SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-Horizon Robotic Manipulation Taskscited by 9
  18. 2019Trajectory Optimization for Unknown Constrained Systems using Reinforcement Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 4; 2024: 7.

15 papers.

Sort:

By area:

  1. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  2. 2022Receding Horizon Inverse Reinforcement Learningcited by 10
  3. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
  4. 2023Learning Reward for Physical Skills using Large Language Modelcited by 5
  5. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedbackcited by 4
  6. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
  7. 2021Off-policy Imitation Learning from Visual Inputscited by 2
  8. 2022Imitation Learning via Differentiable Physicscited by 2
  9. 2022Model-Based Imitation Learning Using Entropy Regularization of Model and Policycited by 1
  10. 2023Signal Temporal Logic-Guided Apprenticeship Learningcited by 1
  11. 2024Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guaranteescited by 1
  12. 2023On the Effective Horizon of Inverse Reinforcement Learningcited by 0
3 more
  1. 2024The Virtues of Pessimism in Inverse Reinforcement Learningcited by 0
  2. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proofcited by 0
  3. 2024On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Active Exploration for Inverse Reinforcement Learning
  2. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  3. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  4. 2025OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learning
  5. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms
  6. 2021Versatile Inverse Reinforcement Learning via Cumulative Rewards
  7. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  8. 2021A Generalised Inverse Reinforcement Learning Framework