mlsanityData to 14 Oct 2026

Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policies

arXiv 2406.18053 ↗First posted 26 Jun 2024cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Hierarchical reinforcement learning (HRL) addresses complex long-horizon tasks by skillfully decomposing them into subgoals. Therefore, the effectiveness of HRL is greatly influenced by subgoal reachability. Typical HRL methods only consider subgoal reachability from the unilateral level, where a dominant level enforces compliance to the subordinate level. However, we observe that when the dominant level becomes trapped in local exploration or generates unattainable subgoals, the subordinate level is negatively affected and cannot follow the dominant level's actions. This can potentially make both levels stuck in local optima, ultimately hindering subsequent subgoal reachability. Allowing real-time bilateral information sharing and error correction would be a natural cure for this issue, which motivates us to propose a mutual response mechanism. Based on this, we propose the Bidirectional-reachable Hierarchical Policy Optimization (BrHPO)--a simple yet effective algorithm that also enjoys computation efficiency. Experiment results on a variety of long-horizon tasks showcase that BrHPO outperforms other state-of-the-art HRL baselines, coupled with a significantly higher exploration efficiency and robustness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 38 of its 56 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

38 of 56 references.

References · 38

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  8. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  9. 2016The Option-Critic Architecturecited by 448
  10. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  11. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  12. 2018Semi-parametric Topological Memory for Navigationcited by 209
26 more
  1. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  2. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  3. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  4. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  5. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  6. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  7. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  8. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  9. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 68
  10. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
  11. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  12. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 58
  13. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 47
  14. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  15. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  16. 2018End-to-End Race Driving with Deep Reinforcement Learningcited by 40
  17. 2020Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learningcited by 29
  18. 2019Learning from Trajectories via Subgoal Discoverycited by 28
  19. 2018Composable Planning with Attributescited by 21
  20. 2021Subgoal Search For Complex Reasoning Taskscited by 21
  21. 2021Hierarchical Reinforcement Learning with Timed Subgoalscited by 21
  22. 2022Intelligent problem-solving as integrated hierarchical reinforcement learningcited by 21
  23. 2022DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learningcited by 19
  24. 2021Active Hierarchical Exploration with Stable Subgoal Representation Learningcited by 17
  25. 2021Adjacency constraint for efficient hierarchical reinforcement learningcited by 10
  26. 2019Inter-Level Cooperation in Hierarchical Reinforcement Learningcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraint
  2. 2024Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
  3. 2024Subgoal-based Hierarchical Reinforcement Learning for Multi-Agent Collaboration ↗
  4. 2026On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning
  5. 2025Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals
  6. 2026Sample Efficient Hierarchical Reinforcement Learning via Best Policy Identification
  7. 2026Hierarchical Reinforcement Learning with Stable Temporal Abstraction for Language Model Agents
  8. 2025Hierarchical Reinforcement Learning with Targeted Causal Interventions