mlsanityData to 7 Oct 2026

Planning for Sample Efficient Imitation Learning

arXiv 2210.09598 ↗First posted 18 Oct 2022cs.LG cs.AIcited by 20

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Imitation learning is a class of promising policy learning algorithms that is free from many practical issues with reinforcement learning, such as the reward design issue and the exploration hardness. However, the current imitation algorithm struggles to achieve both high performance and high in-environment sample efficiency simultaneously. Behavioral Cloning (BC) does not need in-environment interactions, but it suffers from the covariate shift problem which harms its performance. Adversarial Imitation Learning (AIL) turns imitation learning into a distribution matching problem. It can achieve better performance on some tasks but it requires a large number of in-environment interactions. Inspired by the recent success of EfficientZero in RL, we propose EfficientImitate (EI), a planning-based imitation learning method that can achieve high in-environment sample efficiency and performance simultaneously. Our algorithmic contribution in this paper is two-fold. First, we extend AIL into the MCTS-based RL. Second, we show the seemingly incompatible two classes of imitation algorithms (BC and AIL) can be naturally unified under our framework, enjoying the benefits of both. We benchmark our method not only on the state-based DeepMind Control Suite, but also on the image version which many previous works find highly challenging. Experimental results show that EI achieves state-of-the-art results in performance and sample efficiency. EI shows over 4x gain in performance in the limited sample setting on state-based and image-based tasks and can solve challenging problems like Humanoid, where previous methods fail with small amount of interactions. Our code is available at https://github.com/zhaohengyin/EfficientImitate.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 62 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

42 of 62 references.

Closest ancestors · 11

Same direction, newest first

  1. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transport30 Jun
  2. 2022An Empirical Investigation of Representation Learning for Imitation16 May
  3. 2022Rethinking ValueDice: Does It Really Improve Performance?5 Feb
  4. 2021Visual Adversarial Imitation Learning using Variational Models16 Jul
  5. 2021What Matters for Adversarial Imitation Learning?1 Jun
  6. 2021Off-Policy Imitation Learning from Observations25 Feb
  7. 2021MobILE: Model-Based Imitation Learning From Observation Alone22 Feb
  8. 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning2 Oct
  9. 2020Primal Wasserstein Imitation Learning8 Jun
  10. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  11. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 31

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016OpenAI Gymcited by 1,684
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  11. 2020Mastering Atari with Discrete World Modelscited by 990
  12. 2018DeepMind Control Suitecited by 935
19 more
  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  6. 2020Reinforcement Learning with Augmented Datacited by 452
  7. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  9. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  10. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  11. 2021Mastering Atari Games with Limited Datacited by 235
  12. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  13. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  14. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  15. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  16. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  17. 2021Learning and Planning in Complex Action Spacescited by 76
  18. 2020Learning Long-term Visual Dynamics with Region Proposal Interaction Networkscited by 44
  19. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 5; 2025: 7; 2026: 1.

20 papers.

Sort:

By area:

  1. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  2. 2023A Generalist Dynamics Model for Controlcited by 24
  3. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
  4. 2024A Survey of Imitation Learning Methods, Environments and Metricscited by 16
  5. 2024Efficient Imitation Learning with Conservative World Modelscited by 11
  6. 2024Reward-free World Models for Online Imitation Learningcited by 7
  7. 2024Learning to Play Atari in a World of Tokenscited by 6
  8. 2023Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Surveycited by 5
  9. 2023Mimicking Better by Matching the Approximate Action Distributioncited by 5
  10. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  11. 2025Model Predictive Adversarial Imitation Learning for Planning from Observationcited by 2
  12. 2025A KL-regularization Framework for Learning to Plan with Adaptive Priorscited by 2
8 more
  1. 2025Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learningcited by 2
  2. 2023Boosting Reinforcement Learning and Planning with Demonstrations: A Surveycited by 1
  3. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Explorationcited by 1
  4. 2023Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environmentscited by 0
  5. 2025Force-Based Robotic Imitation Learning: A Two-Phase Approach for Construction Assembly Taskscited by 0
  6. 2025Coupled Distributional Random Expert Distillation for World Model Online Imitation Learningcited by 0
  7. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advancescited by 0
  8. 2026Latent Wasserstein Adversarial Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency
  2. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
  3. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms
  4. 2020Augmenting GAIL with BC for sample efficient imitation learning
  5. 2021On Generalization of Adversarial Imitation Learning and Beyond
  6. 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning
  7. 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets
  8. 2023Sample-efficient Adversarial Imitation Learning