mlsanityData to 7 Oct 2026

Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning

arXiv 2110.13625 ↗First posted 26 Oct 2021cs.LGcited by 46

Not on the map. Why?

Abstract (arXiv)

Goal-conditioned hierarchical reinforcement learning (HRL) has shown promising results for solving complex and long-horizon RL tasks. However, the action space of high-level policy in the goal-conditioned HRL is often large, so it results in poor exploration, leading to inefficiency in training. In this paper, we present HIerarchical reinforcement learning Guided by Landmarks (HIGL), a novel framework for training a high-level policy with a reduced action space guided by landmarks, i.e., promising states to explore. The key component of HIGL is twofold: (a) sampling landmarks that are informative for exploration and (b) encouraging the high-level policy to generate a subgoal towards a selected landmark. For (a), we consider two criteria: coverage of the entire visited state space (i.e., dispersion of states) and novelty of states (i.e., prediction error of a state). For (b), we select a landmark as the very first landmark in the shortest path in a graph whose nodes are landmarks. Our experiments demonstrate that our framework outperforms prior-arts across a variety of control tasks, thanks to efficient exploration guided by landmarks.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

41 of 55 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  9. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  10. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  12. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
29 more
  1. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  2. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  3. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  4. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  5. 2017Count-Based Exploration with Neural Density Modelscited by 255
  6. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  7. 2018Semi-parametric Topological Memory for Navigationcited by 203
  8. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  9. 2019Efficient Exploration via State Marginal Matchingcited by 186
  10. 2019Planning with Goal-Conditioned Policiescited by 144
  11. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  12. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  13. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  14. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  15. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  16. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 55
  17. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 53
  18. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  19. 2019Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewardscited by 38
  20. 2020Sparse Graphical Memory for Robust Planningcited by 30
  21. 2020Hallucinative Topological Memory for Zero-Shot Visual Planningcited by 25
  22. 2018Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Explorationcited by 23
  23. 2018Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learningcited by 23
  24. 2018Composable Planning with Attributescited by 21
  25. 2020Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimatecited by 21
  26. 2021Subgoal Search For Complex Reasoning Taskscited by 21
  27. 2017Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learningcited by 15
  28. 2019Learning World Graphs to Accelerate Hierarchical Reinforcement Learningcited by 12
  29. 2018Subgoal Discovery for Hierarchical Dialogue Policy Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 9; 2024: 10; 2025: 11; 2026: 8.

46 papers.

Sort:

By area:

  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  2. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  3. 2025Horizon Reduction Makes RL Scalablecited by 55
  4. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
  5. 2024Machine learning and information theory concepts towards an AI Mathematiciancited by 20
  6. 2022DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learningcited by 18
  7. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14
  8. 2023Imitating Graph-Based Planning with Goal-Conditioned Policiescited by 13
  9. 2022Goal-Space Planning with Subgoal Modelscited by 12
  10. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  11. 2024Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNetscited by 8
  12. 2024Probabilistic Subgoal Representations for Hierarchical Reinforcement learningcited by 8
34 more
  1. 2022Planning Irregular Object Packing via Hierarchical Reinforcement Learningcited by 7
  2. 2023Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigationcited by 6
  3. 2023Hybrid Search for Efficient Planning with Completeness Guaranteescited by 6
  4. 2025Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoalscited by 5
  5. 2025Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planningcited by 5
  6. 2023Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learningcited by 4
  7. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioningcited by 3
  8. 2025Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representationscited by 3
  9. 2023Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rolloutcited by 2
  10. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
  11. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learningcited by 2
  12. 2022Learning user-defined sub-goals using memory editing in reinforcement learningcited by 1
  13. 2022Generalizing to New Tasks via One-Shot Compositional Subgoalscited by 1
  14. 2023Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learningcited by 1
  15. 2024MENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraintcited by 1
  16. 2024SEAL: SEmantic-Augmented Imitation Learning via Language Modelcited by 1
  17. 2024HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Controlcited by 1
  18. 2025Solving Sokoban using Hierarchical Reinforcement Learning with Landmarkscited by 1
  19. 2025Subgoal-Guided Policy Heuristic Search with Learned Subgoalscited by 1
  20. 2025Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learningcited by 1
  21. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  22. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 1
  23. 2022A Fully Controllable Agent in the Path Planning using Goal-Conditioned Reinforcement Learningcited by 0
  24. 2022Multi-Task Option Learning and Discovery for Stochastic Path Planningcited by 0
  25. 2022Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agentscited by 0
  26. 2023Landmark Guided Active Exploration with State-specific Balance Coefficientcited by 0
  27. 2024Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policiescited by 0
  28. 2025Learning with Expert Abstractions for Efficient Multi-Task Continuous Controlcited by 0
  29. 2025Towards better dense rewards in Reinforcement Learning Applicationscited by 0
  30. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Ordercited by 0
  31. 2026Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Explorationcited by 0
  32. 2026S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learningcited by 0
  33. 2026Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learningcited by 0
  34. 2026Metro-WM: Long-Horizon Latent Planning with Realisable Sub-Goalscited by 0