# Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation

arXiv: https://arxiv.org/abs/1905.06750 · HTML: https://mlsanity.com/paper/1905.06750/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 16 May 2019.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 14 of 14)

- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction](https://mlsanity.com/paper/1703.01030/) (2017)

## Built on by (most cited first, 30 of 42)

- [A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](https://mlsanity.com/paper/2309.02473/) (2023)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [f-IRL: Inverse Reinforcement Learning via State Marginal Matching](https://mlsanity.com/paper/2011.04709/) (2020)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Offline Reinforcement Learning as Anti-Exploration](https://mlsanity.com/paper/2106.06431/) (2021)
- [Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees](https://mlsanity.com/paper/2210.01808/) (2022)
- [Energy-Based Imitation Learning](https://mlsanity.com/paper/2004.09395/) (2020)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards](https://mlsanity.com/paper/1905.11108/) (2019)
- [Minimax Optimal Online Imitation Learning via Replay Estimation](https://mlsanity.com/paper/2205.15397/) (2022)
- [Continuous Control with Action Quantization from Demonstrations](https://mlsanity.com/paper/2110.10149/) (2021)
- [On The Fragility of Learned Reward Functions](https://mlsanity.com/paper/2301.03652/) (2023)
- [DITTO: Offline Imitation Learning with World Models](https://mlsanity.com/paper/2302.03086/) (2023)
- [MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations](https://mlsanity.com/paper/2303.17156/) (2023)
- [Hyperparameter Selection for Imitation Learning](https://mlsanity.com/paper/2105.12034/) (2021)
- [Imitation Learning by Reinforcement Learning](https://mlsanity.com/paper/2108.04763/) (2021)
- [Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.16785/) (2020)
- [SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching](https://mlsanity.com/paper/2106.03155/) (2021)
- [Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games](https://mlsanity.com/paper/2106.04958/) (2021)
- [Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2002.06473/) (2020)
- [Imitation Learning from Observations under Transition Model Disparity](https://mlsanity.com/paper/2204.11446/) (2022)
- [A Pragmatic Look at Deep Imitation Learning](https://mlsanity.com/paper/2108.01867/) (2021)
- [Imitation Learning by State-Only Distribution Matching](https://mlsanity.com/paper/2202.04332/) (2022)
- [A Strong Baseline for Batch Imitation Learning](https://mlsanity.com/paper/2302.02788/) (2023)
- [Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learning](https://mlsanity.com/paper/2106.09256/) (2021)
- [Get Back Here: Robust Imitation by Return-to-Distribution Planning](https://mlsanity.com/paper/2305.01400/) (2023)
- [Robust Imitation Learning for Automated Game Testing](https://mlsanity.com/paper/2401.04572/) (2024)
- [Support-weighted Adversarial Imitation Learning](https://mlsanity.com/paper/2002.08803/) (2020)
- [Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency](https://mlsanity.com/paper/2112.06054/) (2021)
- [Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning](https://mlsanity.com/paper/2402.01057/) (2024)

## Abstract (arXiv)

We consider the problem of imitation learning from a finite set of expert trajectories, without access to reinforcement signals. The classical approach of extracting the expert's reward function via inverse reinforcement learning, followed by reinforcement learning is indirect and may be computationally expensive. Recent generative adversarial methods based on matching the policy distribution between the expert and the agent could be unstable during training. We propose a new framework for imitation learning by estimating the support of the expert policy to compute a fixed reward function, which allows us to re-frame imitation learning within the standard reinforcement learning setting. We demonstrate the efficacy of our reward function on both discrete and continuous domains, achieving comparable or better performance than the state of the art under different reinforcement learning algorithms.