mlsanityData to 7 Oct 2026

Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

arXiv 2609.34575 ↗First posted 28 Sep 2026cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline goal-conditioned reinforcement learning (GCRL) learns goal-directed policies from reward-free data, but in long-horizon tasks, goal-conditioned value functions often provide unstable guidance due to sparse rewards and discounting. Hierarchical methods partially mitigate this issue via subgoal decomposition; however, high-level decision-making still relies on noise-sensitive value estimates, leading to unstable behavior in complex environments. We address this limitation by proposing Diffusion Subgoal Planning (DSP), a diffusion-based framework for high-level subgoal generation. DSP casts high-level planning as guided generative inference over goal-conditioned subgoals and learns both conditional and unconditional flows, enabling classifier-free guidance to introduce a goal-directed bias at inference time. By removing explicit value-based guidance from high-level planning, DSP generates reachable and goal-directed subgoals through a generative model while retaining hierarchical execution. Experiments on offline GCRL benchmarks demonstrate that DSP outperforms prior methods on a range of navigation and manipulation tasks, with particularly strong performance in maze environments that require multi-step subgoal planning.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 56 of its 65 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

56 of 65 references.

Closest ancestors · 21

Same direction, newest first

  1. 2026Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL3 Feb
  2. 2025Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning26 Sep
  3. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning8 Sep
  4. 2025Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planning23 Jun
  5. 2025Diffusion Guidance Is a Controllable Policy Improvement Operator29 May
  6. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning19 May
  7. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning16 May
  8. 2025Flow Q-Learning4 Feb
  9. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  10. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL11 Feb
  11. 2023Diffused Task-Agnostic Milestone Planner6 Dec
  12. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning3 Nov
9 more
  1. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models30 Oct
  2. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  3. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?30 May
  4. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  5. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  6. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug
  8. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning24 May
  9. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 35

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Denoising Diffusion Implicit Modelscited by 9,655
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  11. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  12. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
23 more
  1. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  2. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  3. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  4. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  5. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  6. 2024Simple Hierarchical Planning with Diffusioncited by 71
  7. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 69
  8. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 66
  9. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 55
  10. 2025Horizon Reduction Makes RL Scalablecited by 55
  11. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
  12. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  13. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  14. 2022DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learningcited by 18
  15. 2019Learning World Graphs to Accelerate Hierarchical Reinforcement Learningcited by 12
  16. 2023Learning to Reach Goals via Diffusioncited by 10
  17. 2024Subgoal Diffuser: Coarse-to-fine Subgoal Generation to Guide Model Predictive Control for Robot Manipulationcited by 10
  18. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learningcited by 8
  19. 2025Stochastic Interpolants via Conditional Dependent Couplingcited by 6
  20. 2024MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learningcited by 4
  21. 2023Goal-conditioned Offline Planning from Curious Explorationcited by 3
  22. 2025SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planningcited by 1
  23. 2026Drift Flow Matchingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Anchoring Adversarial Trajectories to Data Manifolds: A Bilevel Transfer Optimization Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Flattening Hierarchies with Policy Bootstrapping
  2. 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  3. 2026PathBridger: Subgoal Bridges for Offline Goal-Conditioned Reinforcement Learning
  4. 2025Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals
  5. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
  6. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space
  7. 2026Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusion
  8. 2026Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL