# Curious Hierarchical Actor-Critic Reinforcement Learning

arXiv: https://arxiv.org/abs/2005.03420 · HTML: https://mlsanity.com/paper/2005.03420/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2020.

## Built on by (most cited first, 8 of 8)

- [Survey on reinforcement learning for language processing](https://mlsanity.com/paper/2104.05565/) (2021)
- [Intelligent problem-solving as integrated hierarchical reinforcement learning](https://mlsanity.com/paper/2208.08731/) (2022)
- [Active Hierarchical Exploration with Stable Subgoal Representation Learning](https://mlsanity.com/paper/2105.14750/) (2021)
- [HAC Explore: Accelerating Exploration with Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2108.05872/) (2021)
- [Machine Learning Meets Advanced Robotic Manipulation](https://mlsanity.com/paper/2309.12560/) (2023)
- [Language-Conditioned Reinforcement Learning to Solve Misunderstandings with Action Corrections](https://mlsanity.com/paper/2211.10168/) (2022)
- [Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs](https://mlsanity.com/paper/2307.12063/) (2023)
- [D3HRL: A Distributed Hierarchical Reinforcement Learning Approach Based on Causal Discovery and Spurious Correlation Detection](https://mlsanity.com/paper/2505.01979/) (2025)

## Abstract (arXiv)

Hierarchical abstraction and curiosity-driven exploration are two common paradigms in current reinforcement learning approaches to break down difficult problems into a sequence of simpler ones and to overcome reward sparsity. However, there is a lack of approaches that combine these paradigms, and it is currently unknown whether curiosity also helps to perform the hierarchical abstraction. As a novelty and scientific contribution, we tackle this issue and develop a method that combines hierarchical reinforcement learning with curiosity. Herein, we extend a contemporary hierarchical actor-critic approach with a forward model to develop a hierarchical notion of curiosity. We demonstrate in several continuous-space environments that curiosity can more than double the learning performance and success rates for most of the investigated benchmarking problems. We also provide our source code and a supplementary video.