mlsanityData to 14 Oct 2026

Reconciling Spatial and Temporal Abstractions for Goal Representation

arXiv 2401.09870 ↗First posted 18 Jan 2024cs.LG cs.AI cs.ROcited by 6

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Goal representation affects the performance of Hierarchical Reinforcement Learning (HRL) algorithms by decomposing the complex learning problem into easier subtasks. Recent studies show that representations that preserve temporally abstract environment dynamics are successful in solving difficult problems and provide theoretical guarantees for optimality. These methods however cannot scale to tasks where environment dynamics increase in complexity i.e. the temporally abstract transition relations depend on larger number of variables. On the other hand, other efforts have tried to use spatial abstraction to mitigate the previous issues. Their limitations include scalability to high dimensional environments and dependency on prior knowledge. In this paper, we propose a novel three-layer HRL algorithm that introduces, at different levels of the hierarchy, both a spatial and a temporal goal abstraction. We provide a theoretical study of the regret bounds of the learned policies. We evaluate the approach on complex continuous control tasks, demonstrating the effectiveness of spatial and temporal abstractions learned by this approach. Find open-source code at https://github.com/cosynus-lix/STAR.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 28 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

13 of 28 references.

References · 13

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  3. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  4. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  5. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  6. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  7. 2018Formal Security Analysis of Neural Networks using Symbolic Intervalscited by 185
  8. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  9. 2018Episodic Curiosity through Reachabilitycited by 109
  10. 2018SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planningcited by 78
  11. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 54
  12. 2021Adjacency constraint for efficient hierarchical reinforcement learningcited by 10
1 more
  1. 2023Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysiscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2024Rejecting Hallucinated State Targets during Planningcited by 1
  2. 2025MRS: Multi-Resolution Skills for HRL Agentscited by 0
  3. 2025Learning Multi-Timescale Interventions under Safety and Resource Constraintscited by 0
  4. 2025Can LLMs Translate Human Instructions into a Reinforcement Learning Agent's Internal Emergent Symbolic Representation?cited by 0
  5. 2026Multi-scale Mixture of World Models for Embodied Agents in Evolving Environmentscited by 0
  6. 2026Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis
  2. 2025Learning with Expert Abstractions for Efficient Multi-Task Continuous Control
  3. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learning
  4. 2026Q-Shaped Options for Hierarchical Reinforcement Learning
  5. 2026Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning
  6. 2025Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning
  7. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learning
  8. 2025Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning