mlsanityData to 14 Oct 2026

Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals

arXiv 2505.21750 ↗First posted 27 May 2025cs.LGcited by 5

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Hierarchical reinforcement learning (HRL) learns to make decisions on multiple levels of temporal abstraction. A key challenge in HRL is that the low-level policy changes over time, making it difficult for the high-level policy to generate effective subgoals. To address this issue, the high-level policy must capture a complex subgoal distribution while also accounting for uncertainty in its estimates. We propose an approach that trains a conditional diffusion model regularized by a Gaussian Process (GP) prior to generate a complex variety of subgoals while leveraging principled GP uncertainty quantification. Building on this framework, we develop a strategy that selects subgoals from both the diffusion policy and GP's predictive mean. Our approach outperforms prior HRL methods in both sample efficiency and performance on challenging continuous control benchmarks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 28 of its 45 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

28 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Simple Hierarchical Planning with Diffusion5 Jan
  2. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 25

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  8. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  9. 2016The Option-Critic Architecturecited by 448
  10. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  11. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  12. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
13 more
  1. 2016Variational Intrinsic Controlcited by 267
  2. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  3. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  4. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  5. 2015Gaussian Processes for Data-Efficient Learning in Robotics and Controlcited by 98
  6. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 88
  7. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  8. 2019Exploration via Hindsight Goal Generationcited by 54
  9. 2017Multi-Level Discovery of Deep Optionscited by 51
  10. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 47
  11. 2021Active Hierarchical Exploration with Stable Subgoal Representation Learningcited by 17
  12. 2022State-Conditioned Adversarial Subgoal Generationcited by 8
  13. 2024Probabilistic Subgoal Representations for Hierarchical Reinforcement learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Push Smarter, Not Harder: Hierarchical RL-Diffusion Policy for Efficient Nonprehensile Manipulationcited by 0
  2. 2026Learning Multi-Timescale Abstractions for Hierarchical Combinatorial Planningcited by 0
  3. 2026Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RLcited by 0
  4. 2026S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learningcited by 0
  5. 2026From Connectivity to Rewards: Dense Reward Learning with Directed State Graphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  2. 2026Diffusion Policy through Conditional Proximal Policy Optimization
  3. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning
  4. 2025Overcoming Overfitting in Reinforcement Learning via Gaussian Process Diffusion Policy
  5. 2024Reinforcement Learning with Options and State Representation
  6. 2025Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning
  7. 2026On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning
  8. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning