# Abstract Value Iteration for Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2010.15638 · HTML: https://mlsanity.com/paper/2010.15638/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2020.

## Built on by (most cited first, 12 of 12)

- [Compositional Reinforcement Learning from Logical Specifications](https://mlsanity.com/paper/2106.13906/) (2021)
- [Program Synthesis Guided Reinforcement Learning for Partially Observed Environments](https://mlsanity.com/paper/2102.11137/) (2021)
- [Robust Subtask Learning for Compositional Generalization](https://mlsanity.com/paper/2302.02984/) (2023)
- [CubeTR: Learning to Solve The Rubiks Cube Using Transformers](https://mlsanity.com/paper/2111.06036/) (2021)
- [Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates](https://mlsanity.com/paper/2202.12967/) (2022)
- [Exploiting Multiple Abstractions in Episodic RL via Reward Shaping](https://mlsanity.com/paper/2303.00516/) (2023)
- [Composing Reinforcement Learning Policies, with Formal Guarantees](https://mlsanity.com/paper/2402.13785/) (2024)
- [Achieving Sample and Computational Efficient Reinforcement Learning by Action Space Reduction via Grouping](https://mlsanity.com/paper/2306.12981/) (2023)
- [Accelerating Policy Synthesis in Large-Scale MDPs via Hierarchical Adaptive Refinement](https://mlsanity.com/paper/2506.17792/) (2025)
- [Learning with Expert Abstractions for Efficient Multi-Task Continuous Control](https://mlsanity.com/paper/2503.14809/) (2025)
- [Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines](https://mlsanity.com/paper/2510.27329/) (2025)
- [Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2512.04958/) (2025)

## Abstract (arXiv)

We propose a novel hierarchical reinforcement learning framework for control with continuous state and action spaces. In our framework, the user specifies subgoal regions which are subsets of states; then, we (i) learn options that serve as transitions between these subgoal regions, and (ii) construct a high-level plan in the resulting abstract decision process (ADP). A key challenge is that the ADP may not be Markov, which we address by proposing two algorithms for planning in the ADP. Our first algorithm is conservative, allowing us to prove theoretical guarantees on its performance, which help inform the design of subgoal regions. Our second algorithm is a practical one that interweaves planning at the abstract level and learning at the concrete level. In our experiments, we demonstrate that our approach outperforms state-of-the-art hierarchical reinforcement learning algorithms on several challenging benchmarks.