mlsanityData to 14 Oct 2026

Adversarial Option-Aware Hierarchical Imitation Learning

arXiv 2106.05530 ↗First posted 10 Jun 2021cs.LG cs.ROcited by 19

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

It has been a challenge to learning skills for an agent from long-horizon unannotated demonstrations. Existing approaches like Hierarchical Imitation Learning(HIL) are prone to compounding errors or suboptimal solutions. In this paper, we propose Option-GAIL, a novel method to learn skills at long horizon. The key idea of Option-GAIL is modeling the task hierarchy by options and train the policy via generative adversarial optimization. In particular, we propose an Expectation-Maximization(EM)-style algorithm: an E-step that samples the options of expert conditioned on the current learned policy, and an M-step that updates the low- and high-level policies of agent simultaneously to minimize the newly proposed option-occupancy measurement between the expert and the agent. We theoretically prove the convergence of the proposed algorithm. Experiments show that Option-GAIL outperforms other counterparts consistently across a variety of tasks.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 15 of its 32 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

15 of 32 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Provable Hierarchical Imitation Learning via EM7 Oct
  2. 2020Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets19 May
  3. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  4. 2018Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information29 Sep

Other references · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016OpenAI Gymcited by 1,736
  3. 2016Generative Adversarial Imitation Learningcited by 1,597
  4. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  5. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  6. 2016The Option-Critic Architecturecited by 448
  7. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  8. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  9. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  10. 2017OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learningcited by 40
  11. 2018One-Shot Hierarchical Imitation Learning of Compound Visuomotor Taskscited by 33

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 7; 2024: 4; 2025: 2.

19 papers.

Sort:

By area:

  1. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  2. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  3. 2022Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Controlcited by 16
  4. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  5. 2023Multi-task Hierarchical Adversarial Inverse Reinforcement Learningcited by 15
  6. 2022Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Taskscited by 10
  7. 2022Transfering Hierarchical Structure with Dual Meta Imitation Learningcited by 7
  8. 2022A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Processcited by 7
  9. 2023Automated Task-Time Interventions to Improve Teamwork using Imitation Learningcited by 7
  10. 2023GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimationcited by 6
  11. 2022SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environmentscited by 5
  12. 2024IDIL: Imitation Learning of Intent-Driven Expert Behaviorcited by 5
7 more
  1. 2023Hierarchical Deep Counterfactual Regret Minimizationcited by 4
  2. 2024Identifying Selections for Unsupervised Subtask Discoverycited by 4
  3. 2021Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learningcited by 3
  4. 2025Hierarchical Imitation Learning of Team Behavior from Heterogeneous Demonstrationscited by 2
  5. 2024SEAL: SEmantic-Augmented Imitation Learning via Language Modelcited by 1
  6. 2023CasIL: Cognizing and Imitating Skills via a Dual Cognition-Action Architecturecited by 0
  7. 2025Intention-aware Hierarchical Diffusion Model for Long-term Trajectory Anomaly Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective
  2. 2020On Computation and Generalization of Generative Adversarial Imitation Learning
  3. 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets
  4. 2019Hindsight Generative Adversarial Imitation Learning
  5. 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning
  6. 2021Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
  7. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
  8. 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms