# Sample-Efficient Imitation Learning via Generative Adversarial Nets

arXiv: https://arxiv.org/abs/1809.02064 · HTML: https://mlsanity.com/paper/1809.02064/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 34 of its 57 references that are in the map.
- First posted on arXiv 6 Sep 2018.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [Energy-based Generative Adversarial Network](https://mlsanity.com/paper/1609.03126/) (2016)
- [Learning values across many orders of magnitude](https://mlsanity.com/paper/1602.07714/) (2016)
- [Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic](https://mlsanity.com/paper/1611.02247/) (2016)
- [Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets](https://mlsanity.com/paper/1705.10479/) (2017)
- [Model-Free Imitation Learning with Policy Optimization](https://mlsanity.com/paper/1605.08478/) (2016)

## Built on by (most cited first, 30 of 36)

- [ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations](https://mlsanity.com/paper/2107.14483/) (2021)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [State Alignment-based Imitation Learning](https://mlsanity.com/paper/1911.10947/) (2019)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Task-Relevant Adversarial Imitation Learning](https://mlsanity.com/paper/1910.01077/) (2019)
- [Visual Adversarial Imitation Learning using Variational Models](https://mlsanity.com/paper/2107.08829/) (2021)
- [Positive-Unlabeled Reward Learning](https://mlsanity.com/paper/1911.00459/) (2019)
- [Imitation Learning with Sinkhorn Distances](https://mlsanity.com/paper/2008.09167/) (2020)
- [Online Apprenticeship Learning](https://mlsanity.com/paper/2102.06924/) (2021)
- [Learning by Watching](https://mlsanity.com/paper/2106.05966/) (2021)
- [Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.16785/) (2020)
- [OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching](https://mlsanity.com/paper/2109.04307/) (2021)
- [A Survey of Demonstration Learning](https://mlsanity.com/paper/2303.11191/) (2023)
- [Adversarial Imitation Learning from Visual Observations using Latent Information](https://mlsanity.com/paper/2309.17371/) (2023)
- [Efficient Imitation Learning with Conservative World Models](https://mlsanity.com/paper/2405.13193/) (2024)
- [A Pragmatic Look at Deep Imitation Learning](https://mlsanity.com/paper/2108.01867/) (2021)
- [Visually Robust Adversarial Imitation Learning from Videos with Contrastive Learning](https://mlsanity.com/paper/2407.12792/) (2024)
- [Goal-Aware Generative Adversarial Imitation Learning from Imperfect Demonstration for Robotic Cloth Manipulation](https://mlsanity.com/paper/2209.10149/) (2022)
- [Forward and inverse reinforcement learning sharing network weights and hyperparameters](https://mlsanity.com/paper/2008.07284/) (2020)
- [Mimicking Better by Matching the Approximate Action Distribution](https://mlsanity.com/paper/2306.09805/) (2023)
- [Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective](https://mlsanity.com/paper/2312.11214/) (2023)
- [Latent Policies for Adversarial Imitation Learning](https://mlsanity.com/paper/2206.11299/) (2022)
- [Deep Reinforcement Learning with Explicit Context Representation](https://mlsanity.com/paper/2310.09924/) (2023)
- [Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning](https://mlsanity.com/paper/2310.19831/) (2023)
- [Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shaping](https://mlsanity.com/paper/2410.03847/) (2024)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)
- [Conditional Kernel Imitation Learning for Continuous State Environments](https://mlsanity.com/paper/2308.12573/) (2023)
- [Sample Efficient Imitation Learning via Reward Function Trained in Advance](https://mlsanity.com/paper/2111.11711/) (2021)
- [Model-Based Imitation Learning Using Entropy Regularization of Model and Policy](https://mlsanity.com/paper/2206.10101/) (2022)

## Abstract (arXiv)

GAIL is a recent successful imitation learning architecture that exploits the adversarial training procedure introduced in GANs. Albeit successful at generating behaviours similar to those demonstrated to the agent, GAIL suffers from a high sample complexity in the number of interactions it has to carry out in the environment in order to achieve satisfactory performance. We dramatically shrink the amount of interactions with the environment necessary to learn well-behaved imitation policies, by up to several orders of magnitude. Our framework, operating in the model-free regime, exhibits a significant increase in sample-efficiency over previous methods by simultaneously a) learning a self-tuned adversarially-trained surrogate reward and b) leveraging an off-policy actor-critic architecture. We show that our approach is simple to implement and that the learned agents remain remarkably stable, as shown in our experiments that span a variety of continuous control tasks. Video visualisations available at: urlhttps://youtu.be/-nCsqUJnRKU.