# Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis

arXiv: https://arxiv.org/abs/2309.07168 · HTML: https://mlsanity.com/paper/2309.07168/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 10 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 8 of 8)

- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2006.11485/) (2020)
- [Symbol Emergence in Cognitive Developmental Systems: a Survey](https://mlsanity.com/paper/1801.08829/) (2018)
- [Hierarchical Affordance Discovery using Intrinsic Motivation](https://mlsanity.com/paper/2009.10968/) (2020)
- [Learning a Set of Interrelated Tasks by Using Sequences of Motor Policies for a Strategic Intrinsically Motivated Learner](https://mlsanity.com/paper/1810.04877/) (2018)
- [Robots Learn Increasingly Complex Tasks with Intrinsic Motivation and Automatic Curriculum Learning](https://mlsanity.com/paper/2202.10222/) (2022)

## Abstract (arXiv)

Open-ended learning benefits immensely from the use of symbolic methods for goal representation as they offer ways to structure knowledge for efficient and transferable learning. However, the existing Hierarchical Reinforcement Learning (HRL) approaches relying on symbolic reasoning are often limited as they require a manual goal representation. The challenge in autonomously discovering a symbolic goal representation is that it must preserve critical information, such as the environment dynamics. In this work, we propose a developmental mechanism for subgoal discovery via an emergent representation that abstracts (i.e., groups together) sets of environment states that have similar roles in the task. We create a HRL algorithm that gradually learns this representation along with the policies and evaluate it on navigation tasks to show the learned representation is interpretable and results in data efficiency.