# Planning for Sample Efficient Imitation Learning arXiv: https://arxiv.org/abs/2210.09598 ยท HTML: https://mlsanity.com/paper/2210.09598/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. - Builds on 42 of its 62 references that are in the map. - First posted on arXiv 18 Oct 2022. ## Placed in - Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/) ## Builds on (most cited first, 30 of 42) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015) - [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018) - [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020) - [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018) - [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016) - [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019) - [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016) - [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019) - [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018) - [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020) - [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018) - [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019) - [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020) - [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017) - [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020) - [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017) - [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020) - [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022) - [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021) - [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017) - [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020) - [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021) - [Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model](https://mlsanity.com/paper/1907.00953/) (2019) - [DeepMDP: Learning Continuous Latent Space Models for Representation Learning](https://mlsanity.com/paper/1906.02736/) (2019) - [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018) - [DART: Noise Injection for Robust Imitation Learning](https://mlsanity.com/paper/1703.09327/) (2017) - [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019) - [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020) - [Pretraining Representations for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2106.04799/) (2021) ## Built on by (most cited first, 20 of 20) - [Imitation Bootstrapped Reinforcement Learning](https://mlsanity.com/paper/2311.02198/) (2023) - [A Generalist Dynamics Model for Control](https://mlsanity.com/paper/2305.10912/) (2023) - [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://mlsanity.com/paper/2310.08348/) (2023) - [A Survey of Imitation Learning Methods, Environments and Metrics](https://mlsanity.com/paper/2404.19456/) (2024) - [Efficient Imitation Learning with Conservative World Models](https://mlsanity.com/paper/2405.13193/) (2024) - [Reward-free World Models for Online Imitation Learning](https://mlsanity.com/paper/2410.14081/) (2024) - [Learning to Play Atari in a World of Tokens](https://mlsanity.com/paper/2406.01361/) (2024) - [Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey](https://mlsanity.com/paper/2304.12090/) (2023) - [Mimicking Better by Matching the Approximate Action Distribution](https://mlsanity.com/paper/2306.09805/) (2023) - [Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shaping](https://mlsanity.com/paper/2410.03847/) (2024) - [Model Predictive Adversarial Imitation Learning for Planning from Observation](https://mlsanity.com/paper/2507.21533/) (2025) - [A KL-regularization Framework for Learning to Plan with Adaptive Priors](https://mlsanity.com/paper/2510.04280/) (2025) - [Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learning](https://mlsanity.com/paper/2510.09487/) (2025) - [Boosting Reinforcement Learning and Planning with Demonstrations: A Survey](https://mlsanity.com/paper/2303.13489/) (2023) - [Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration](https://mlsanity.com/paper/2506.20307/) (2025) - [Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environments](https://mlsanity.com/paper/2310.05712/) (2023) - [Force-Based Robotic Imitation Learning: A Two-Phase Approach for Construction Assembly Tasks](https://mlsanity.com/paper/2501.14942/) (2025) - [Coupled Distributional Random Expert Distillation for World Model Online Imitation Learning](https://mlsanity.com/paper/2505.02228/) (2025) - [Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances](https://mlsanity.com/paper/2511.03565/) (2025) - [Latent Wasserstein Adversarial Imitation Learning](https://mlsanity.com/paper/2603.05440/) (2026) ## Abstract (arXiv) Imitation learning is a class of promising policy learning algorithms that is free from many practical issues with reinforcement learning, such as the reward design issue and the exploration hardness. However, the current imitation algorithm struggles to achieve both high performance and high in-environment sample efficiency simultaneously. Behavioral Cloning (BC) does not need in-environment interactions, but it suffers from the covariate shift problem which harms its performance. Adversarial Imitation Learning (AIL) turns imitation learning into a distribution matching problem. It can achieve better performance on some tasks but it requires a large number of in-environment interactions. Inspired by the recent success of EfficientZero in RL, we propose EfficientImitate (EI), a planning-based imitation learning method that can achieve high in-environment sample efficiency and performance simultaneously. Our algorithmic contribution in this paper is two-fold. First, we extend AIL into the MCTS-based RL. Second, we show the seemingly incompatible two classes of imitation algorithms (BC and AIL) can be naturally unified under our framework, enjoying the benefits of both. We benchmark our method not only on the state-based DeepMind Control Suite, but also on the image version which many previous works find highly challenging. Experimental results show that EI achieves state-of-the-art results in performance and sample efficiency. EI shows over 4x gain in performance in the limited sample setting on state-based and image-based tasks and can solve challenging problems like Humanoid, where previous methods fail with small amount of interactions. Our code is available at https://github.com/zhaohengyin/EfficientImitate.