mlsanityData to 7 Oct 2026

Simple Hierarchical Planning with Diffusion

arXiv 2401.02644 ↗First posted 5 Jan 2024cs.LG cs.AIcited by 71

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion-based generative methods have proven effective in modeling trajectories with offline datasets. However, they often face computational challenges and can falter in generalization, especially in capturing temporal abstractions for long-horizon tasks. To overcome this, we introduce the Hierarchical Diffuser, a simple, fast, yet surprisingly effective planning method combining the advantages of hierarchical and diffusion-based planning. Our model adopts a "jumpy" planning strategy at the higher level, which allows it to have a larger receptive field but at a lower computational cost -- a crucial factor for diffusion-based planning methods, as we have empirically verified. Additionally, the jumpy sub-goals guide our low-level planner, facilitating a fine-tuning stage and further improving our approach's effectiveness. We conducted empirical evaluations on standard offline reinforcement learning benchmarks, demonstrating our method's superior performance and efficiency in terms of training and planning speed compared to the non-hierarchical Diffuser as well as other hierarchical planning methods. Moreover, we explore our model's generalization capability, particularly on how our method improves generalization capabilities on compositional out-of-distribution tasks.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 50 of its 62 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

50 of 62 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  2. 2023EDGI: Equivariant Diffusion for Planning with Embodied Agents22 Mar
  3. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  4. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 45

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  12. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
33 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2018World Modelscited by 1,444
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  13. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  14. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  15. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  16. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  17. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  18. 2022Deep Hierarchical Planning from Pixelscited by 96
  19. 2019Exploring Model-based Planning with Policy Networkscited by 90
  20. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  21. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 66
  22. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 60
  23. 2020On the role of planning in model-based deep reinforcement learningcited by 57
  24. 2022Skill-based Model-based Reinforcement Learningcited by 50
  25. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  26. 2021Clockwork Variational Autoencoderscited by 40
  27. 2019Variational Temporal Abstractioncited by 37
  28. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  29. 2023Planning Goals for Explorationcited by 28
  30. 2022Robustness Implies Generalization via Data-Dependent Generalization Boundscited by 23
  31. 2022Mastering the Unsupervised Reinforcement Learning Benchmark from Pixelscited by 20
  32. 2020Latent Skill Planning for Exploration and Transfercited by 19
  33. 2023Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domainscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 40; 2026: 18.

71 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 162
  2. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  3. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  4. 2024PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Modelscited by 41
  5. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 40
  6. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  7. 2025What Makes a Good Diffusion Planner for Decision Making?cited by 37
  8. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 34
  9. 2025Monte Carlo Tree Diffusion for System 2 Planningcited by 33
  10. 2024Autoregressive Action Sequence Learning for Robotic Manipulationcited by 26
  11. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 24
  12. 2024Generative Image as Action Modelscited by 18
38 more
  1. 2024Diffusion Model for Planning: A Systematic Literature Reviewcited by 17
  2. 2025Reinforcement Learning for Flow-Matching Policiescited by 14
  3. 2024Resisting Stochastic Risks in Diffusion Planners with the Trajectory Aggregation Treecited by 13
  4. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learningcited by 13
  5. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  6. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 11
  7. 2025SafeFlowMatcher: Safe and Fast Planning using Flow Matching with Control Barrier Functionscited by 11
  8. 2025TeLoGraF: Temporal Logic Planning via Graph-encoded Flow Matchingcited by 9
  9. 2025Multi-agent Coordination via Flow Matchingcited by 9
  10. 2025Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planningcited by 8
  11. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 8
  12. 2025Unleashing Flow Policies with Distributional Criticscited by 7
  13. 2024Solving Continual Offline RL through Selective Weights Activation on Aligned Spacescited by 6
  14. 2025CHD: Coupled Hierarchical Diffusion for Long-Horizon Taskscited by 6
  15. 2025Fast Monte Carlo Tree Diffusion: 100x Speedup via Parallel Sparse Planningcited by 6
  16. 2025Hybrid Diffusion for Simultaneous Symbolic and Continuous Planningcited by 6
  17. 2026Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusioncited by 6
  18. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  19. 2025Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Rankingcited by 5
  20. 2025Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoalscited by 5
  21. 2026Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RLcited by 5
  22. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learningcited by 4
  23. 2025Exploratory Diffusion Model for Unsupervised Reinforcement Learningcited by 4
  24. 2025Cascaded Diffusion Models for Neural Motion Planningcited by 4
  25. 2025Tree-Guided Diffusion Plannercited by 4
  26. 2025Extremum Flow Matching for Offline Goal Conditioned Reinforcement Learningcited by 3
  27. 2025Train-Once Plan-Anywhere Kinodynamic Motion Planning via Diffusion Treescited by 3
  28. 2025Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learningcited by 3
  29. 2025Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learningcited by 2
  30. 2025Consistency Trajectory Planning: High-Quality and Efficient Trajectory Optimization for Offline Model-Based Reinforcement Learningcited by 2
  31. 2025Compositional Monte Carlo Tree Diffusion for Extendable Planningcited by 2
  32. 2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Controlcited by 2
  33. 2026Compositional Visual Planning via Inference-Time Diffusion Scalingcited by 2
  34. 2026Refining Compositional Diffusion for Reliable Long-Horizon Planningcited by 2
  35. 2026HDFlow: Hierarchical Diffusion-Flow Planning for Long-horizon Taskscited by 2
  36. 2026Unifying Object-Centric World Models and Diffusion Policy: A Hierarchical Framework for Multi-Stage Robotic Taskscited by 2
  37. 2026Reversal Q-Learningcited by 2
  38. 2024What Matters in Hierarchical Search for Combinatorial Reasoning Problems?cited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RL
  2. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning
  3. 2023Diffused Task-Agnostic Milestone Planner
  4. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL
  5. 2026Amortizing Trajectory Diffusion with Keyed Drift Fields
  6. 2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalization
  7. 2024Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner
  8. 2025Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making