# Exploring Hierarchy-Aware Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/1807.05037 · HTML: https://mlsanity.com/paper/1807.05037/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 3 of its 22 references that are in the map.
- First posted on arXiv 13 Jul 2018.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 3 of 3)

- [Learning the Preferences of Ignorant, Inconsistent Agents](https://mlsanity.com/paper/1512.05832/) (2015)
- [Modeling Human Understanding of Complex Intentional Action with a Bayesian Nonparametric Subgoal Model](https://mlsanity.com/paper/1512.00964/) (2015)
- [An Alternative Softmax Operator for Reinforcement Learning](https://mlsanity.com/paper/1612.05628/) (2016)

## Abstract (arXiv)

We introduce a new generative model for human planning under the Bayesian Inverse Reinforcement Learning (BIRL) framework which takes into account the fact that humans often plan using hierarchical strategies. We describe the Bayesian Inverse Hierarchical RL (BIHRL) algorithm for inferring the values of hierarchical planners, and use an illustrative toy model to show that BIHRL retains accuracy where standard BIRL fails. Furthermore, BIHRL is able to accurately predict the goals of `Wikispeedia' game players, with inclusion of hierarchical structure in the model resulting in a large boost in accuracy. We show that BIHRL is able to significantly outperform BIRL even when we only have a weak prior on the hierarchical structure of the plans available to the agent, and discuss the significant challenges that remain for scaling up this framework to more realistic settings.