mlsanityData to 14 Oct 2026

Active Hierarchical Exploration with Stable Subgoal Representation Learning

arXiv 2105.14750 ↗First posted 31 May 2021cs.LG cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Goal-conditioned hierarchical reinforcement learning (GCHRL) provides a promising approach to solving long-horizon tasks. Recently, its success has been extended to more general settings by concurrently learning hierarchical policies and subgoal representations. Although GCHRL possesses superior exploration ability by decomposing tasks via subgoals, existing GCHRL methods struggle in temporally extended tasks with sparse external rewards, since the high-level policy learning relies on external rewards. As the high-level policy selects subgoals in an online learned representation space, the dynamic change of the subgoal space severely hinders effective high-level exploration. In this paper, we propose a novel regularization that contributes to both stable and efficient subgoal representation learning. Building upon the stable representation, we design measures of novelty and potential for subgoals, and develop an active hierarchical exploration strategy that seeks out new promising subgoals and states without intrinsic rewards. Experimental results show that our approach significantly outperforms state-of-the-art baselines in continuous control tasks with sparse rewards.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

38 of 51 references.

References · 38

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  8. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  9. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  10. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  11. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  12. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
26 more
  1. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  2. 2016Variational Intrinsic Controlcited by 267
  3. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  4. 2017Count-Based Exploration with Neural Density Modelscited by 257
  5. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  6. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  7. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
  8. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  9. 2018Count-Based Exploration with the Successor Representationcited by 112
  10. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  11. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  12. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  13. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  14. 2016Deep Successor Reinforcement Learningcited by 72
  15. 2013Empowerment -- an Introductioncited by 68
  16. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 68
  17. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  18. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 54
  19. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  20. 2019Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewardscited by 40
  21. 2019Generalization of Reinforcement Learners with Working and Episodic Memorycited by 36
  22. 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
  23. 2018Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Explorationcited by 23
  24. 2017Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learningcited by 15
  25. 2019Hierarchical Policy Learning is Sensitive to Goal Space Designcited by 8
  26. 2020Curious Hierarchical Actor-Critic Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 4; 2024: 6; 2025: 3; 2026: 2.

17 papers.

Sort:

By area:

  1. 2024Probabilistic Subgoal Representations for Hierarchical Reinforcement learningcited by 8
  2. 2021DisTop: Discovering a Topological representation to learn diverse and rewarding skillscited by 7
  3. 2025Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoalscited by 5
  4. 2023Learning to Solve Tasks with Exploring Prior Behaviourscited by 4
  5. 2025Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representationscited by 3
  6. 2022Generalizing to New Tasks via One-Shot Compositional Subgoalscited by 1
  7. 2023Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphscited by 1
  8. 2024Scaling Artificial Intelligence for Digital Wargaming in Support of Decision-Makingcited by 1
  9. 2024HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Controlcited by 1
  10. 2024Generalization of Compositional Tasks with Logical Specification via Implicit Planningcited by 1
  11. 2025Subgoal-Guided Policy Heuristic Search with Learned Subgoalscited by 1
  12. 2023Landmark Guided Active Exploration with State-specific Balance Coefficientcited by 0
5 more
  1. 2023Policy Optimization with Smooth Guidance Learned from State-Only Demonstrationscited by 0
  2. 2024Scaling Intelligent Agents in Combat Simulations for Wargamingcited by 0
  3. 2024Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policiescited by 0
  4. 2026S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learningcited by 0
  5. 2026From Connectivity to Rewards: Dense Reward Learning with Directed State Graphscited by 0