# Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information

arXiv: https://arxiv.org/abs/1810.01266 · HTML: https://mlsanity.com/paper/1810.01266/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 29 Sep 2018.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Built on by (most cited first, 30 of 41)

- [Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives](https://mlsanity.com/paper/2303.09824/) (2023)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [Meta-Inverse Reinforcement Learning with Probabilistic Context Variables](https://mlsanity.com/paper/1909.09314/) (2019)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics](https://mlsanity.com/paper/2211.02231/) (2022)
- [Learning Options via Compression](https://mlsanity.com/paper/2212.04590/) (2022)
- [Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Information](https://mlsanity.com/paper/2103.07084/) (2021)
- [Adversarial Option-Aware Hierarchical Imitation Learning](https://mlsanity.com/paper/2106.05530/) (2021)
- [Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control](https://mlsanity.com/paper/2210.01969/) (2022)
- [Hierarchical Imitation Learning with Vector Quantized Models](https://mlsanity.com/paper/2301.12962/) (2023)
- [Multi-task Hierarchical Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/2305.12633/) (2023)
- [Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks](https://mlsanity.com/paper/2107.05825/) (2021)
- [Motion Planning by Learning the Solution Manifold in Trajectory Optimization](https://mlsanity.com/paper/2107.05842/) (2021)
- [Inverse Contextual Bandits: Learning How Behavior Evolves over Time](https://mlsanity.com/paper/2107.06317/) (2021)
- [Language-guided Skill Learning with Temporal Variational Inference](https://mlsanity.com/paper/2402.16354/) (2024)
- [Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Tasks](https://mlsanity.com/paper/2301.00051/) (2022)
- [From Simple to Complex Skills: The Case of In-Hand Object Reorientation](https://mlsanity.com/paper/2501.05439/) (2025)
- [Domain-Adversarial and Conditional State Space Model for Imitation Learning](https://mlsanity.com/paper/2001.11628/) (2020)
- [Transfering Hierarchical Structure with Dual Meta Imitation Learning](https://mlsanity.com/paper/2201.11981/) (2022)
- [Automated Task-Time Interventions to Improve Teamwork using Imitation Learning](https://mlsanity.com/paper/2303.00413/) (2023)
- [GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2312.10802/) (2023)
- [Bayesian Nonparametrics for Offline Skill Discovery](https://mlsanity.com/paper/2202.04675/) (2022)
- [SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environments](https://mlsanity.com/paper/2204.01922/) (2022)
- [Learning Multi-Task Transferable Rewards via Variational Inverse Reinforcement Learning](https://mlsanity.com/paper/2206.09498/) (2022)
- [BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human Racing Gameplay](https://mlsanity.com/paper/2402.14194/) (2024)
- [IDIL: Imitation Learning of Intent-Driven Expert Behavior](https://mlsanity.com/paper/2404.16989/) (2024)
- [Stabilizing Long-term Multi-turn Reinforcement Learning with Gated Rewards](https://mlsanity.com/paper/2508.10548/) (2025)
- [Deep Probabilistic Movement Primitives with a Bayesian Aggregator](https://mlsanity.com/paper/2307.05141/) (2023)
- [Identifying Selections for Unsupervised Subtask Discovery](https://mlsanity.com/paper/2410.21616/) (2024)
- [oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions](https://mlsanity.com/paper/2002.09043/) (2020)

## Abstract (arXiv)

The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. In this work, we discover the interaction between sub-tasks from their resulting state-action trajectory sequences using a directed graphical model. We propose a new algorithm based on the generative adversarial imitation learning framework which automatically learns sub-task policies from unsegmented demonstrations. Our approach maximizes the directed information flow in the graphical model between sub-task latent variables and their generated trajectories. We also show how our approach connects with the existing Options framework, which is commonly used to learn hierarchical policies.