mlsanityData to 7 Oct 2026

Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rollout

arXiv 2309.13508 ↗First posted 24 Sep 2023cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Goal-conditioned hierarchical reinforcement learning (HRL) presents a promising approach for enabling effective exploration in complex, long-horizon reinforcement learning (RL) tasks through temporal abstraction. Empirically, heightened inter-level communication and coordination can induce more stable and robust policy improvement in hierarchical systems. Yet, most existing goal-conditioned HRL algorithms have primarily focused on the subgoal discovery, neglecting inter-level cooperation. Here, we propose a goal-conditioned HRL framework named Guided Cooperation via Model-based Rollout (GCMR), aiming to bridge inter-layer information synchronization and cooperation by exploiting forward dynamics. Firstly, the GCMR mitigates the state-transition error within off-policy correction via model-based rollout, thereby enhancing sample efficiency. Secondly, to prevent disruption by the unseen subgoals and states, lower-level Q-function gradients are constrained using a gradient penalty with a model-inferred upper bound, leading to a more stable behavioral policy conducive to effective exploration. Thirdly, we propose a one-step rollout-based planning, using higher-level critics to guide the lower-level policy. Specifically, we estimate the value of future states of the lower-level policy using the higher-level critic function, thereby transmitting global task information downwards to avoid local pitfalls. These three critical components in GCMR are expected to facilitate inter-level cooperation significantly. Experimental results demonstrate that incorporating the proposed GCMR framework with a disentangled variant of HIGL, namely ACLG, yields more stable and robust policy improvement compared to various baselines and significantly outperforms previous state-of-the-art algorithms.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 47 of its 72 references that are in the map. First posted on arXiv 24 Sep 2023.

Builds on

47 of 72 references.

References · 47

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2017Improved Training of Wasserstein GANscited by 2,223
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  10. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  12. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
35 more
  1. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  2. 2018Semi-parametric Topological Memory for Navigationcited by 203
  3. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  4. 2020Revisiting Fundamentals of Experience Replaycited by 171
  5. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  6. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  7. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  8. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  9. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  10. 2020BADGR: An Autonomous Self-Supervised Learning-Based Navigation Systemcited by 97
  11. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  12. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  13. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  14. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  15. 2022A Survey on Model-based Reinforcement Learningcited by 65
  16. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 55
  17. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
  18. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  19. 2015Learning Deep Control Policies for Autonomous Aerial Vehicles with MPC-Guided Policy Searchcited by 41
  20. 2020Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learningcited by 34
  21. 2020Sparse Graphical Memory for Robust Planningcited by 30
  22. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  23. 2020Hallucinative Topological Memory for Zero-Shot Visual Planningcited by 25
  24. 2020LaND: Learning to Navigate from Disengagementscited by 24
  25. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  26. 2020Plan2Vec: Unsupervised Representation Learning by Latent Planscited by 21
  27. 2022DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learningcited by 18
  28. 2020Model-based Policy Optimization with Unsupervised Model Adaptationcited by 16
  29. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  30. 2021MapGo: Model-Assisted Policy Optimization for Goal-Oriented Taskscited by 13
  31. 2023Imitating Graph-Based Planning with Goal-Conditioned Policiescited by 13
  32. 2021Adjacency constraint for efficient hierarchical reinforcement learningcited by 10
  33. 2015Model-based Reinforcement Learning with Parametrized Physical Models and Optimism-Driven Explorationcited by 8
  34. 2022State-Conditioned Adversarial Subgoal Generationcited by 8
  35. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxationcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNetscited by 8
  2. 2024HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Controlcited by 1