# Globally Optimal Hierarchical Reinforcement Learning for Linearly-Solvable Markov Decision Processes

arXiv: https://arxiv.org/abs/2106.15380 · HTML: https://mlsanity.com/paper/2106.15380/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 29 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Composable Deep Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/1803.06773/) (2018)
- [Munchausen Reinforcement Learning](https://mlsanity.com/paper/2007.14430/) (2020)
- [The Option Keyboard: Combining Skills in Reinforcement Learning](https://mlsanity.com/paper/2106.13105/) (2021)
- [A Boolean Task Algebra for Reinforcement Learning](https://mlsanity.com/paper/2001.01394/) (2020)
- [Logistic Q-Learning](https://mlsanity.com/paper/2010.11151/) (2020)
- [Policy Iteration for Factored MDPs](https://mlsanity.com/paper/1301.3869/) (2013)
- [Hierarchical Linearly-Solvable Markov Decision Problems](https://mlsanity.com/paper/1603.03267/) (2016)
- [Fast rates for online learning in Linearly Solvable Markov Decision Processes](https://mlsanity.com/paper/1702.06341/) (2017)

## Built on by (most cited first, 6 of 6)

- [Dyadic Reinforcement Learning](https://mlsanity.com/paper/2308.07843/) (2023)
- [Hierarchical Average-Reward Linearly-solvable Markov Decision Processes](https://mlsanity.com/paper/2407.06690/) (2024)
- [A Unified Theory of Compositionality, Modularity, and Interpretability in Markov Decision Processes](https://mlsanity.com/paper/2506.09499/) (2025)
- [Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2512.04958/) (2025)
- [The Terminal Representation in Reinforcement Learning](https://mlsanity.com/paper/2605.31289/) (2026)
- [A Goal-Set Characterization of Task Composition in the Boolean Task Algebra](https://mlsanity.com/paper/2606.04053/) (2026)

## Abstract (arXiv)

In this work we present a novel approach to hierarchical reinforcement learning for linearly-solvable Markov decision processes. Our approach assumes that the state space is partitioned, and the subtasks consist in moving between the partitions. We represent value functions on several levels of abstraction, and use the compositionality of subtasks to estimate the optimal values of the states in each partition. The policy is implicitly defined on these optimal value estimates, rather than being decomposed among the subtasks. As a consequence, our approach can learn the globally optimal policy, and does not suffer from the non-stationarity of high-level decisions. If several partitions have equivalent dynamics, the subtasks of those partitions can be shared. If the set of boundary states is smaller than the entire state space, our approach can have significantly smaller sample complexity than that of a flat learner, and we validate this empirically in several experiments.