mlsanityData to 14 Oct 2026

Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning

arXiv 2506.14045 ↗First posted 16 Jun 2025cs.AIcited by 22

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Developing agents capable of exploring, planning and learning in complex open-ended environments is a grand challenge in artificial intelligence (AI). Hierarchical reinforcement learning (HRL) offers a promising solution to this challenge by discovering and exploiting the temporal structure within a stream of experience. The strong appeal of the HRL framework has led to a rich and diverse body of literature attempting to discover a useful structure. However, it is still not clear how one might define what constitutes good structure in the first place, or the kind of problems in which identifying it may be helpful. This work aims to identify the benefits of HRL from the perspective of the fundamental challenges in decision-making, as well as highlight its impact on the performance trade-offs of AI agents. Through these benefits, we then cover the families of methods that discover temporal structure in HRL, ranging from learning directly from online experience to offline datasets, to leveraging large language models (LLMs). Finally, we highlight the challenges of temporal structure discovery and the domains that are particularly well-suited for such endeavours.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. First posted on arXiv 16 Jun 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 19.

22 papers.

Sort:

By area:

  1. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agentscited by 20
  2. 2026HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agentscited by 18
  3. 2026Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusioncited by 7
  4. 2026Compositional Planning with Jumpy World Modelscited by 7
  5. 2025Scalable Option Learning in High-Throughput Environmentscited by 5
  6. 2025ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learningcited by 5
  7. 2026Laplacian Representations for Decision-Time Planningcited by 5
  8. 2026Adaptive Time Series Reasoning via Segment Selectioncited by 4
  9. 2026Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discoverycited by 1
  10. 2026Enabling Option Learning in Sparse Rewards with Hindsight Experience Replaycited by 0
  11. 2026Auction-Based Online Policy Adaptation for Evolving Objectivescited by 0
  12. 2026Hierarchical Behaviour Spacescited by 0
10 more
  1. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0
  2. 2026Goal-Conditioned Agents that Learn Everything All at Oncecited by 0
  3. 2026An Information-Theoretic Definition for Open-Ended Learningcited by 0
  4. 2026Event-Driven Reinforcement Learning Enables Long-Horizon Control in Semiconductor Fabricationcited by 0
  5. 2026Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Executioncited by 0
  6. 2026Darwin Mobile Agent: A Roadmap for Self-Evolutioncited by 0
  7. 2026Compositional Behavioral Semantics for State Abstraction in Reinforcement Learningcited by 0
  8. 2026Mesh-RL: Coupled subgrid reinforcement learningcited by 0
  9. 2026Up and Down the Abstraction Ladder: Code-Based Skills for Language Agentscited by 0
  10. 2026Mastering Atari 2600 Games with Discovered Optionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Hierarchical Decision Making Based on Structural Information Principles
  2. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learning
  3. 2024Reconciling Spatial and Temporal Abstractions for Goal Representation
  4. 2026Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents
  5. 2026Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents
  6. 2026Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search
  7. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
  8. 2023Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis