# Provable Hierarchical Imitation Learning via EM

arXiv: https://arxiv.org/abs/2010.03133 · HTML: https://mlsanity.com/paper/2010.03133/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 18 of its 50 references that are in the map.
- First posted on arXiv 7 Oct 2020.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 18 of 18)

- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Statistical guarantees for the EM algorithm: From population to sample-based analysis](https://mlsanity.com/paper/1408.2156/) (2014)
- [Non-convex Optimization for Machine Learning](https://mlsanity.com/paper/1712.07897/) (2017)
- [A Deep Hierarchical Approach to Lifelong Learning in Minecraft](https://mlsanity.com/paper/1604.07255/) (2016)
- [Hierarchical Imitation and Reinforcement Learning](https://mlsanity.com/paper/1803.00590/) (2018)
- [TACO: Learning Task Decomposition via Temporal Alignment for Control](https://mlsanity.com/paper/1803.01840/) (2018)
- [Multi-Level Discovery of Deep Options](https://mlsanity.com/paper/1703.08294/) (2017)
- [DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations](https://mlsanity.com/paper/1710.05421/) (2017)
- [Ten Steps of EM Suffice for Mixtures of Two Gaussians](https://mlsanity.com/paper/1609.00368/) (2016)
- [Singularity, Misspecification, and the Convergence Rate of EM](https://mlsanity.com/paper/1810.00828/) (2018)
- [Global analysis of Expectation Maximization for mixtures of two Gaussians](https://mlsanity.com/paper/1608.07630/) (2016)
- [Regularized EM Algorithms: A Unified Framework and Statistical Guarantees](https://mlsanity.com/paper/1511.08551/) (2015)
- [Exploration--Exploitation in MDPs with Options](https://mlsanity.com/paper/1703.08667/) (2017)
- [Statistical and Computational Guarantees for the Baum-Welch Algorithm](https://mlsanity.com/paper/1512.08269/) (2015)
- [On learning parametric-output HMMs](https://mlsanity.com/paper/1302.6009/) (2013)

## Built on by (most cited first, 10 of 10)

- [Adversarial Option-Aware Hierarchical Imitation Learning](https://mlsanity.com/paper/2106.05530/) (2021)
- [Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control](https://mlsanity.com/paper/2210.01969/) (2022)
- [Hierarchical Imitation Learning with Vector Quantized Models](https://mlsanity.com/paper/2301.12962/) (2023)
- [IDIL: Imitation Learning of Intent-Driven Expert Behavior](https://mlsanity.com/paper/2404.16989/) (2024)
- [Online Baum-Welch algorithm for Hierarchical Imitation Learning](https://mlsanity.com/paper/2103.12197/) (2021)
- [Hierarchical Imitation Learning of Team Behavior from Heterogeneous Demonstrations](https://mlsanity.com/paper/2502.17618/) (2025)
- [Offline Imitation of Badminton Player Behavior via Experiential Contexts and Brownian Motion](https://mlsanity.com/paper/2403.12406/) (2024)
- [SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environments](https://mlsanity.com/paper/2407.18913/) (2024)
- [CasIL: Cognizing and Imitating Skills via a Dual Cognition-Action Architecture](https://mlsanity.com/paper/2309.16299/) (2023)
- [Interpretable Robotic Manipulation from Language](https://mlsanity.com/paper/2405.17047/) (2024)

## Abstract (arXiv)

Due to recent empirical successes, the options framework for hierarchical reinforcement learning is gaining increasing popularity. Rather than learning from rewards which suffers from the curse of dimensionality, we consider learning an options-type hierarchical policy from expert demonstrations. Such a problem is referred to as hierarchical imitation learning. Converting this problem to parameter inference in a latent variable model, we theoretically characterize the EM approach proposed by Daniel et al. (2016). The population level algorithm is analyzed as an intermediate step, which is nontrivial due to the samples being correlated. If the expert policy can be parameterized by a variant of the options framework, then under regularity conditions, we prove that the proposed algorithm converges with high probability to a norm ball around the true parameter. To our knowledge, this is the first performance guarantee for an hierarchical imitation learning algorithm that only observes primitive state-action pairs.