mlsanityData to 7 Oct 2026

Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning

arXiv 2006.11485 ↗First posted 20 Jun 2020cs.LG stat.MLcited by 55

Not on the map. Why?

Abstract (arXiv)

Goal-conditioned hierarchical reinforcement learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is often large. Searching in a large goal space poses difficulties for both high-level subgoal generation and low-level policy learning. In this paper, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a k-step adjacent region of the current state using an adjacency constraint. We theoretically prove that the proposed adjacency constraint preserves the optimal hierarchical policy in deterministic MDPs, and show that this constraint can be practically implemented by training an adjacency network that can discriminate between adjacent and non-adjacent subgoals. Experimental results on discrete and continuous control tasks show that incorporating the adjacency constraint improves the performance of state-of-the-art HRL approaches in both deterministic and stochastic environments.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 10; 2023: 13; 2024: 10; 2025: 11; 2026: 6.

55 papers.

Sort:

By area:

  1. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  2. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  3. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
  4. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  5. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
  6. 2022Intelligent problem-solving as integrated hierarchical reinforcement learningcited by 21
  7. 2021Hierarchical Reinforcement Learning with Timed Subgoalscited by 20
  8. 2022DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learningcited by 18
  9. 2021Active Hierarchical Exploration with Stable Subgoal Representation Learningcited by 17
  10. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14
  11. 2023Imitating Graph-Based Planning with Goal-Conditioned Policiescited by 13
  12. 2022Goal-Space Planning with Subgoal Modelscited by 12
38 more
  1. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  2. 2021Adjacency constraint for efficient hierarchical reinforcement learningcited by 10
  3. 2022Reachability-Aware Laplacian Representation in Reinforcement Learningcited by 10
  4. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 10
  5. 2022Continual Auxiliary Task Learningcited by 9
  6. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learningcited by 9
  7. 2022State-Conditioned Adversarial Subgoal Generationcited by 8
  8. 2024Probabilistic Subgoal Representations for Hierarchical Reinforcement learningcited by 8
  9. 2025Flattening Hierarchies with Policy Bootstrappingcited by 8
  10. 2025Structured Reinforcement Learning for Combinatorial Decision-Makingcited by 7
  11. 2023Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigationcited by 6
  12. 2023Goal Space Abstraction in Hierarchical Reinforcement Learning via Set-Based Reachability Analysiscited by 6
  13. 2024TopoNav: Topological Navigation for Efficient Exploration in Sparse Reward Environmentscited by 5
  14. 2024Stochastic Q-learning for Large Discrete Action Spacescited by 5
  15. 2025Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoalscited by 5
  16. 2025Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planningcited by 5
  17. 2023Dynamic Neighborhood Construction for Structured Large Discrete Action Spacescited by 4
  18. 2024Variable-Agnostic Causal Exploration for Reinforcement Learningcited by 4
  19. 2021Shortest-Path Constrained Reinforcement Learning for Sparse Reward Taskscited by 2
  20. 2023PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learningcited by 2
  21. 2023Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rolloutcited by 2
  22. 2024The intrinsic motivation of reinforcement and imitation learning for sequential taskscited by 2
  23. 2022Generalizing to New Tasks via One-Shot Compositional Subgoalscited by 1
  24. 2023Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learningcited by 1
  25. 2023Hierarchical Task Network Planning for Facilitating Cooperative Multi-Agent Reinforcement Learningcited by 1
  26. 2023Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphscited by 1
  27. 2024HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Controlcited by 1
  28. 2025Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learningcited by 1
  29. 2025DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learningcited by 1
  30. 2026Action-Sufficient Goal Representationscited by 1
  31. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 1
  32. 2022SHIRO: Soft Hierarchical Reinforcement Learningcited by 0
  33. 2023Landmark Guided Active Exploration with State-specific Balance Coefficientcited by 0
  34. 2023Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysiscited by 0
  35. 2024Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policiescited by 0
  36. 2024Safe Policy Exploration Improvement via Subgoalscited by 0
  37. 2025MRS: Multi-Resolution Skills for HRL Agentscited by 0
  38. 2025Hierarchical Reinforcement Learning with Targeted Causal Interventionscited by 0

50 most cited of 55 All 55 (JSON)