mlsanityData to 7 Oct 2026

Online Baum-Welch algorithm for Hierarchical Imitation Learning

arXiv 2103.12197 ↗First posted 22 Mar 2021cs.LG cs.SY eess.SYcited by 3

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

The options framework for hierarchical reinforcement learning has increased its popularity in recent years and has made improvements in tackling the scalability problem in reinforcement learning. Yet, most of these recent successes are linked with a proper options initialization or discovery. When an expert is available, the options discovery problem can be addressed by learning an options-type hierarchical policy directly from expert demonstrations. This problem is referred to as hierarchical imitation learning and can be handled as an inference problem in a Hidden Markov Model, which is done via an Expectation-Maximization type algorithm. In this work, we propose a novel online algorithm to perform hierarchical imitation learning in the options framework. Further, we discuss the benefits of such an algorithm and compare it with its batch version in classical reinforcement learning benchmarks. We show that this approach works well in both discrete and continuous environments and, under certain conditions, it outperforms the batch version.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 31 references that are in the map. First posted on arXiv 22 Mar 2021.

Builds on

17 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Provable Hierarchical Imitation Learning via EM7 Oct
  2. 2018Fast Policy Learning through Imitation and Reinforcement26 May

Other references · 15

  1. 2016OpenAI Gymcited by 1,684
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  3. 2016The Option-Critic Architecturecited by 433
  4. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  5. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  6. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  7. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  8. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  9. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  10. 2016Learning and Transfer of Modulated Locomotor Controllerscited by 59
  11. 2017Multi-Level Discovery of Deep Optionscited by 50
  12. 2017OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learningcited by 40
3 more
  1. 2018One-Shot Hierarchical Imitation Learning of Compound Visuomotor Taskscited by 32
  2. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 29
  3. 2016HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewardscited by 15

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
  2. 2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learningcited by 6
  3. 2024SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environmentscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2018Accelerating Imitation Learning with Predictive Models
  2. 2022Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control
  3. 2023A Strong Baseline for Batch Imitation Learning
  4. 2021Adversarial Option-Aware Hierarchical Imitation Learning
  5. 2022On Efficient Online Imitation Learning via Classification
  6. 2021Stochastic convex optimization for provably efficient apprenticeship learning
  7. 2024Reward-free World Models for Online Imitation Learning
  8. 2023Hierarchical Imitation Learning with Vector Quantized Models