# Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2506.14045 · HTML: https://mlsanity.com/paper/2506.14045/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- First posted on arXiv 16 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Built on by (most cited first, 22 of 22)

- [Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents](https://mlsanity.com/paper/2509.03581/) (2025)
- [HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents](https://mlsanity.com/paper/2602.16165/) (2026)
- [Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusion](https://mlsanity.com/paper/2602.02722/) (2026)
- [Compositional Planning with Jumpy World Models](https://mlsanity.com/paper/2602.19634/) (2026)
- [Scalable Option Learning in High-Throughput Environments](https://mlsanity.com/paper/2509.00338/) (2025)
- [ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning](https://mlsanity.com/paper/2510.14176/) (2025)
- [Laplacian Representations for Decision-Time Planning](https://mlsanity.com/paper/2602.05031/) (2026)
- [Adaptive Time Series Reasoning via Segment Selection](https://mlsanity.com/paper/2602.18645/) (2026)
- [Going Beyond State-Reaching: Learning Abstractions for Intrinsically Motivated Option Discovery](https://mlsanity.com/paper/2609.36473/) (2026)
- [Enabling Option Learning in Sparse Rewards with Hindsight Experience Replay](https://mlsanity.com/paper/2602.13865/) (2026)
- [Auction-Based Online Policy Adaptation for Evolving Objectives](https://mlsanity.com/paper/2604.02151/) (2026)
- [Hierarchical Behaviour Spaces](https://mlsanity.com/paper/2604.24558/) (2026)
- [Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning](https://mlsanity.com/paper/2605.13207/) (2026)
- [Goal-Conditioned Agents that Learn Everything All at Once](https://mlsanity.com/paper/2605.23551/) (2026)
- [An Information-Theoretic Definition for Open-Ended Learning](https://mlsanity.com/paper/2606.08369/) (2026)
- [Event-Driven Reinforcement Learning Enables Long-Horizon Control in Semiconductor Fabrication](https://mlsanity.com/paper/2606.10705/) (2026)
- [Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution](https://mlsanity.com/paper/2606.20014/) (2026)
- [Darwin Mobile Agent: A Roadmap for Self-Evolution](https://mlsanity.com/paper/2606.20622/) (2026)
- [Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning](https://mlsanity.com/paper/2606.25357/) (2026)
- [Mesh-RL: Coupled subgrid reinforcement learning](https://mlsanity.com/paper/2606.26333/) (2026)
- [Up and Down the Abstraction Ladder: Code-Based Skills for Language Agents](https://mlsanity.com/paper/2609.31076/) (2026)
- [Mastering Atari 2600 Games with Discovered Options](https://mlsanity.com/paper/2610.03604/) (2026)

## Abstract (arXiv)

Developing agents capable of exploring, planning and learning in complex open-ended environments is a grand challenge in artificial intelligence (AI). Hierarchical reinforcement learning (HRL) offers a promising solution to this challenge by discovering and exploiting the temporal structure within a stream of experience. The strong appeal of the HRL framework has led to a rich and diverse body of literature attempting to discover a useful structure. However, it is still not clear how one might define what constitutes good structure in the first place, or the kind of problems in which identifying it may be helpful. This work aims to identify the benefits of HRL from the perspective of the fundamental challenges in decision-making, as well as highlight its impact on the performance trade-offs of AI agents. Through these benefits, we then cover the families of methods that discover temporal structure in HRL, ranging from learning directly from online experience to offline datasets, to leveraging large language models (LLMs). Finally, we highlight the challenges of temporal structure discovery and the domains that are particularly well-suited for such endeavours.