# HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewards

arXiv: https://arxiv.org/abs/1604.06508 · HTML: https://mlsanity.com/paper/1604.06508/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2016.

## Built on by (most cited first, 15 of 15)

- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Active Task-Inference-Guided Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/2001.09227/) (2020)
- [Recurrent Reasoning with Vision-Language Models for Estimating Long-Horizon Embodied Task Progress](https://mlsanity.com/paper/2603.17312/) (2026)
- [SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environments](https://mlsanity.com/paper/2204.01922/) (2022)
- [Planning on the fast lane: Learning to interact using attention mechanisms in path integral inverse reinforcement learning](https://mlsanity.com/paper/2007.05798/) (2020)
- [Inverse Reinforcement Learning via Nonparametric Spatio-Temporal Subgoal Modeling](https://mlsanity.com/paper/1803.00444/) (2018)
- [Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning](https://mlsanity.com/paper/1806.08479/) (2018)
- [oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions](https://mlsanity.com/paper/2002.09043/) (2020)
- [Online Baum-Welch algorithm for Hierarchical Imitation Learning](https://mlsanity.com/paper/2103.12197/) (2021)
- [Option Compatible Reward Inverse Reinforcement Learning](https://mlsanity.com/paper/1911.02723/) (2019)
- [A Generalized Apprenticeship Learning Framework for Capturing Evolving Student Pedagogical Strategies](https://mlsanity.com/paper/2602.20527/) (2026)
- [An Offline Time-aware Apprenticeship Learning Framework for Evolving Reward Functions](https://mlsanity.com/paper/2305.09070/) (2023)
- [Context-Hierarchy Inverse Reinforcement Learning](https://mlsanity.com/paper/2202.12597/) (2022)
- [Inverse Delayed Reinforcement Learning](https://mlsanity.com/paper/2412.02931/) (2024)
- [Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards](https://mlsanity.com/paper/2604.00258/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) struggles in problems with delayed rewards, and one approach is to segment the task into sub-tasks with incremental rewards. We propose a framework called Hierarchical Inverse Reinforcement Learning (HIRL), which is a model for learning sub-task structure from demonstrations. HIRL decomposes the task into sub-tasks based on transitions that are consistent across demonstrations. These transitions are defined as changes in local linearity w.r.t to a kernel function. Then, HIRL uses the inferred structure to learn reward functions local to the sub-tasks but also handle any global dependencies such as sequentiality. We have evaluated HIRL on several standard RL benchmarks: Parallel Parking with noisy dynamics, Two-Link Pendulum, 2D Noisy Motion Planning, and a Pinball environment. In the parallel parking task, we find that rewards constructed with HIRL converge to a policy with an 80% success rate in 32% fewer time-steps than those constructed with Maximum Entropy Inverse RL (MaxEnt IRL), and with partial state observation, the policies learned with IRL fail to achieve this accuracy while HIRL still converges. We further find that that the rewards learned with HIRL are robust to environment noise where they can tolerate 1 stdev. of random perturbation in the poses in the environment obstacles while maintaining roughly the same convergence rate. We find that HIRL rewards can converge up-to 6x faster than rewards constructed with IRL.