mlsanityData to 14 Oct 2026

DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning

arXiv 2210.05150 ↗First posted 11 Oct 2022cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Hierarchical Reinforcement Learning (HRL) has made notable progress in complex control tasks by leveraging temporal abstraction. However, previous HRL algorithms often suffer from serious data inefficiency as environments get large. The extended components, i.e., goal space and length of episodes, impose a burden on either one or both high-level and low-level policies since both levels share the total horizon of the episode. In this paper, we present a method of Decoupling Horizons Using a Graph in Hierarchical Reinforcement Learning (DHRL) which can alleviate this problem by decoupling the horizons of high-level and low-level policies and bridging the gap between the length of both horizons using a graph. DHRL provides a freely stretchable high-level action interval, which facilitates longer temporal abstraction and faster training in complex tasks. Our method outperforms state-of-the-art HRL algorithms in typical HRL environments. Moreover, DHRL achieves long and complex locomotion and manipulation tasks.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

22 of 31 references.

References · 22

  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  7. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  8. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  9. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  10. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  11. 2018Learning to Walk via Deep Reinforcement Learningcited by 158
  12. 2019Planning with Goal-Conditioned Policiescited by 149
10 more
  1. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  2. 2017Control of a Quadrotor with Reinforcement Learningcited by 92
  3. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 69
  4. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  5. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 47
  6. 2019Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learningcited by 44
  7. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  8. 2019Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Realcited by 42
  9. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 30
  10. 2019Hierarchical Reinforcement Learning for Quadruped Locomotioncited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 6; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  2. 2023CQM: Curriculum Reinforcement Learning with a Quantized World Modelcited by 11
  3. 2025Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representationscited by 3
  4. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 3
  5. 2023Machine Learning Meets Advanced Robotic Manipulationcited by 2
  6. 2023Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rolloutcited by 2
  7. 2024What Matters in Hierarchical Search for Combinatorial Reasoning Problems?cited by 1
  8. 2024HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Controlcited by 1
  9. 2025Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learningcited by 1
  10. 2026Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agentscited by 1
  11. 2024Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policiescited by 0
  12. 2025DHP: Discrete Hierarchical Planning for Hierarchical Reinforcement Learning Agentscited by 0
7 more
  1. 2025A tale of two goals: leveraging sequentiality in multi-goal scenarioscited by 0
  2. 2025Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learningcited by 0
  3. 2026Learning Transferable Skills in Action RPGs via Directed Skill Graphs and Selective Adaptationcited by 0
  4. 2026E2DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulationcited by 0
  5. 2026Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systemscited by 0
  6. 2026From Connectivity to Rewards: Dense Reward Learning with Directed State Graphscited by 0
  7. 2026Metro-WM: Long-Horizon Latent Planning with Realisable Sub-Goalscited by 0