mlsanityData to 14 Oct 2026

Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysis

arXiv 2309.07675 ↗First posted 14 Sep 2023cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Open-ended learning benefits immensely from the use of symbolic methods for goal representation as they offer ways to structure knowledge for efficient and transferable learning. However, the existing Hierarchical Reinforcement Learning (HRL) approaches relying on symbolic reasoning are often limited as they require a manual goal representation. The challenge in autonomously discovering a symbolic goal representation is that it must preserve critical information, such as the environment dynamics. In this paper, we propose a developmental mechanism for goal discovery via an emergent representation that abstracts (i.e., groups together) sets of environment states that have similar roles in the task. We introduce a Feudal HRL algorithm that concurrently learns both the goal representation and a hierarchical policy. The algorithm uses symbolic reachability analysis for neural networks to approximate the transition relation among sets of states and to refine the goal representation. We evaluate our approach on complex navigation tasks, showing the learned representation is interpretable, transferrable and results in data efficient learning.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

18 of 33 references.

References · 18

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  4. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  5. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  6. 2019Algorithms for Verifying Deep Neural Networkscited by 221
  7. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  8. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  9. 2018Episodic Curiosity through Reachabilitycited by 109
  10. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 79
  11. 2018SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planningcited by 78
  12. 2020Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Explorationcited by 68
6 more
  1. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  2. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 54
  3. 2016Towards Deep Symbolic Reinforcement Learningcited by 42
  4. 2018Symbol Emergence in Cognitive Developmental Systems: a Surveycited by 39
  5. 2021Learning Symbolic Rules for Interpretable Deep Reinforcement Learningcited by 9
  6. 2018Learning a Set of Interrelated Tasks by Using Sequences of Motor Policies for a Strategic Intrinsically Motivated Learnercited by 4

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 2.

6 papers.

  1. 2024Reconciling Spatial and Temporal Abstractions for Goal Representationcited by 6
  2. 2024The intrinsic motivation of reinforcement and imitation learning for sequential taskscited by 2
  3. 2024Sample Efficient Reinforcement Learning by Automatically Learning to Compose Subtaskscited by 0
  4. 2024Unsupervised Learning of Effective Actions in Roboticscited by 0
  5. 2025Proposing Hierarchical Goal-Conditioned Policy Planning in Multi-Goal Reinforcement Learningcited by 0
  6. 2025Can LLMs Translate Human Instructions into a Reinforcement Learning Agent's Internal Emergent Symbolic Representation?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis
  2. 2024Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning
  3. 2025Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning
  4. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learning
  5. 2024Hierarchical Decision Making Based on Structural Information Principles
  6. 2025Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robots
  7. 2025Learning with Expert Abstractions for Efficient Multi-Task Continuous Control
  8. 2024Reinforcement Learning with Options and State Representation