mlsanityData to 7 Oct 2026

A Pragmatic Look at Deep Imitation Learning

arXiv 2108.01867 ↗First posted 4 Aug 2021cs.LG cs.NE stat.MLcited by 9

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

The introduction of the generative adversarial imitation learning (GAIL) algorithm has spurred the development of scalable imitation learning approaches using deep neural networks. Many of the algorithms that followed used a similar procedure, combining on-policy actor-critic algorithms with inverse reinforcement learning. More recently there have been an even larger breadth of approaches, most of which use off-policy algorithms. However, with the breadth of algorithms, everything from datasets to base reinforcement learning algorithms to evaluation settings can vary, making it difficult to fairly compare them. In this work we re-implement 6 different IL algorithms, updating 3 of them to be off-policy, base them on a common off-policy algorithm (SAC), and evaluate them on a widely-used expert trajectory dataset (D4RL) for the most common benchmark (MuJoCo). After giving all algorithms the same hyperparameter optimisation budget, we compare their results for a range of expert trajectories. In summary, GAIL, with all of its improvements, consistently performs well across a range of sample sizes, AdRIL is a simple contender that performs well with one important hyperparameter to tune, and behavioural cloning remains a strong baseline when data is more plentiful.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 76 references that are in the map. First posted on arXiv 4 Aug 2021.

Builds on

40 of 76 references.

Closest ancestors · 10

Same direction, newest first

  1. 2021What Matters for Adversarial Imitation Learning?1 Jun
  2. 2021Hyperparameter Selection for Imitation Learning25 May
  3. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap4 Mar
  4. 2020Addressing reward bias in Adversarial Imitation Learning with neutral reward functions20 Sep
  5. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning28 Jun
  6. 2020Primal Wasserstein Imitation Learning8 Jun
  7. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  8. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation16 May
  9. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
  10. 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets6 Sep

Other references · 30

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2017Improved Training of Wasserstein GANscited by 2,223
  8. 2016OpenAI Gymcited by 1,684
  9. 2016Generative Adversarial Imitation Learningcited by 1,576
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  11. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  12. 2018Exploration by Random Network Distillationcited by 966
18 more
  1. 2017Deep Reinforcement Learning that Matterscited by 727
  2. 2018Which Training Methods for GANs do actually Converge?cited by 672
  3. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  4. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  5. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  6. 2020A Metric Learning Reality Checkcited by 317
  7. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  8. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  9. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  10. 2015Generative Moment Matching Networkscited by 252
  11. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  12. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167
  13. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  14. 2017Mind the Class Weight Bias: Weighted Maximum Mean Discrepancy for Unsupervised Domain Adaptationcited by 151
  15. 2015Training generative neural networks via Maximum Mean Discrepancy optimizationcited by 149
  16. 2017Time Limits in Reinforcement Learningcited by 72
  17. 2019Positive-Unlabeled Reward Learningcited by 36
  18. 2020Batch Exploration with Examples for Scalable Robotic Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 1; 2026: 3.

9 papers.

  1. 2024A Comparison of Imitation Learning Algorithms for Bimanual Manipulationcited by 8
  2. 2023Learning with Constraint Learning: New Perspective, Solution Strategy and Various Applicationscited by 4
  3. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  4. 2024Automating Deformable Gasket Assemblycited by 1
  5. 2024Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrationscited by 1
  6. 2026Learning from Demonstrations via Capability-Aware Goal Samplingcited by 1
  7. 2025Force-Based Robotic Imitation Learning: A Two-Phase Approach for Construction Assembly Taskscited by 0
  8. 2026Latent Wasserstein Adversarial Imitation Learningcited by 0
  9. 2026Disagreement-Regularized Imitation Learning for Image-Based Continuous Control with Gaussian and Beta Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020On Computation and Generalization of Generative Adversarial Imitation Learning
  2. 2025Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization
  3. 2020Augmenting GAIL with BC for sample efficient imitation learning
  4. 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergence
  5. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances
  6. 2023Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective
  7. 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning
  8. 2024C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory