mlsanityData to 14 Oct 2026

Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Tasks

arXiv 2301.00051 ↗First posted 30 Dec 2022cs.LG cs.AI cs.ROcited by 10

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Adversarial imitation learning (AIL) has become a popular alternative to supervised imitation learning that reduces the distribution shift suffered by the latter. However, AIL requires effective exploration during an online reinforcement learning phase. In this work, we show that the standard, naive approach to exploration can manifest as a suboptimal local maximum if a policy learned with AIL sufficiently matches the expert distribution without fully learning the desired task. This can be particularly catastrophic for manipulation tasks, where the difference between an expert and a non-expert state-action pair is often subtle. We present Learning from Guided Play (LfGP), a framework in which we leverage expert demonstrations of multiple exploratory, auxiliary tasks in addition to a main task. The addition of these auxiliary tasks forces the agent to explore states and actions that standard AIL may learn to ignore. Additionally, this particular formulation allows for the reusability of expert data between main tasks. Our experimental results in a challenging multitask robotic manipulation domain indicate that LfGP significantly outperforms both AIL and behaviour cloning, while also being more expert sample efficient than these baselines. To explain this performance gap, we provide further analysis of a toy problem that highlights the coupling between a local maximum and poor exploration, and also visualize the differences between the learned models from AIL and LfGP.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 54 references that are in the map. First posted on arXiv 30 Dec 2022.

Builds on

37 of 54 references.

Closest ancestors · 8

Same direction, newest first

  1. 2021Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning16 Dec
  2. 2021Adversarial Option-Aware Hierarchical Imitation Learning10 Jun
  3. 2021What Matters for Adversarial Imitation Learning?1 Jun
  4. 2021Hyperparameter Selection for Imitation Learning25 May
  5. 2020Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning1 Jul
  6. 2019Task-Relevant Adversarial Imitation Learning2 Oct
  7. 2018Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information29 Sep
  8. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2017Improved Training of Wasserstein GANscited by 2,235
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2016Generative Adversarial Imitation Learningcited by 1,597
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
17 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  2. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  3. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  4. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  5. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  6. 2019Learning Latent Plans from Playcited by 316
  7. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  8. 2020First return, then explorecited by 266
  9. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  10. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 95
  11. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 69
  12. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 68
  13. 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
  14. 2017OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learningcited by 40
  15. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Modelscited by 14
  16. 2017The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneouslycited by 5
  17. 2021Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 0; 2026: 2.

10 papers.

  1. 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 17
  2. 2023Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensorcited by 9
  3. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  4. 2023A Statistical Guarantee for Representation Transfer in Multitask Imitation Learningcited by 1
  5. 2024AUTONODE: A Neuro-Graphic Self-Learnable Engine for Cognitive GUI Automationcited by 1
  6. 2024Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examplescited by 1
  7. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learningcited by 1
  8. 2024State-of-the-art in Robot Learning for Multi-Robot Collaboration: A Comprehensive Surveycited by 0
  9. 2024Learning the Generalizable Manipulation Skills on Soft-body Tasks via Guided Self-attention Behavior Cloning Policycited by 0
  10. 2026How to Mitigate the Distribution Shift Problem in Robotics Control: A Robust and Adaptive Approach Based on Offline to Online Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
  2. 2024Efficient Imitation Learning with Conservative World Models
  3. 2019A New Framework for Query Efficient Active Imitation Learning
  4. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
  5. 2021Imitation Learning by Reinforcement Learning
  6. 2020ADAIL: Adaptive Adversarial Imitation Learning
  7. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning
  8. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning