# Hierarchical Linearly-Solvable Markov Decision Problems

arXiv: https://arxiv.org/abs/1603.03267 · HTML: https://mlsanity.com/paper/1603.03267/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2016.

## Built on by (most cited first, 3 of 3)

- [Globally Optimal Hierarchical Reinforcement Learning for Linearly-Solvable Markov Decision Processes](https://mlsanity.com/paper/2106.15380/) (2021)
- [Hierarchical Average-Reward Linearly-solvable Markov Decision Processes](https://mlsanity.com/paper/2407.06690/) (2024)
- [The Terminal Representation in Reinforcement Learning](https://mlsanity.com/paper/2605.31289/) (2026)

## Abstract (arXiv)

We present a hierarchical reinforcement learning framework that formulates each task in the hierarchy as a special type of Markov decision process for which the Bellman equation is linear and has analytical solution. Problems of this type, called linearly-solvable MDPs (LMDPs) have interesting properties that can be exploited in a hierarchical setting, such as efficient learning of the optimal value function or task compositionality. The proposed hierarchical approach can also be seen as a novel alternative to solving LMDPs with large state spaces. We derive a hierarchical version of the so-called Z-learning algorithm that learns different tasks simultaneously and show empirically that it significantly outperforms the state-of-the-art learning methods in two classical hierarchical reinforcement learning domains: the taxi domain and an autonomous guided vehicle task.