# What Makes a Good Diffusion Planner for Decision Making?

arXiv: https://arxiv.org/abs/2503.00535 · HTML: https://mlsanity.com/paper/2503.00535/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 44 of its 57 references that are in the map.
- First posted on arXiv 1 Mar 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Offline RL and imitation learning from data / [Diffusion and flow policies for offline RL](https://mlsanity.com/offlinerl/diffusion-and-flow-policies-for-offline-rl/)

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies](https://mlsanity.com/paper/2304.10573/) (2023)
- [Imitating Human Behaviour with Diffusion Models](https://mlsanity.com/paper/2301.10677/) (2023)
- [Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling](https://mlsanity.com/paper/2209.14548/) (2022)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners](https://mlsanity.com/paper/2302.01877/) (2023)

## Built on by (most cited first, 30 of 38)

- [Generative Trajectory Stitching through Diffusion Composition](https://mlsanity.com/paper/2503.05153/) (2025)
- [Prior-Guided Diffusion Planning for Offline Reinforcement Learning](https://mlsanity.com/paper/2505.10881/) (2025)
- [Enhanced DACER Algorithm with High Diffusion Efficiency](https://mlsanity.com/paper/2505.23426/) (2025)
- [State-Covering Trajectory Stitching for Diffusion Planners](https://mlsanity.com/paper/2506.00895/) (2025)
- [One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning](https://mlsanity.com/paper/2508.13904/) (2025)
- [Multi-agent Coordination via Flow Matching](https://mlsanity.com/paper/2511.05005/) (2025)
- [Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planning](https://mlsanity.com/paper/2506.00867/) (2025)
- [Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusion](https://mlsanity.com/paper/2602.02722/) (2026)
- [Fast Monte Carlo Tree Diffusion: 100x Speedup via Parallel Sparse Planning](https://mlsanity.com/paper/2506.09498/) (2025)
- [GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models](https://mlsanity.com/paper/2605.29398/) (2026)
- [Tree-Guided Diffusion Planner](https://mlsanity.com/paper/2508.21800/) (2025)
- [MaskedManipulator: Versatile Whole-Body Manipulation](https://mlsanity.com/paper/2505.19086/) (2025)
- [Extremum Flow Matching for Offline Goal Conditioned Reinforcement Learning](https://mlsanity.com/paper/2505.19717/) (2025)
- [Implicit Maximum Likelihood Estimation for Real-time Generative Model Predictive Control](https://mlsanity.com/paper/2603.13733/) (2026)
- [Consistency Trajectory Planning: High-Quality and Efficient Trajectory Optimization for Offline Model-Based Reinforcement Learning](https://mlsanity.com/paper/2507.09534/) (2025)
- [Compositional Monte Carlo Tree Diffusion for Extendable Planning](https://mlsanity.com/paper/2510.21361/) (2025)
- [Compositional Visual Planning via Inference-Time Diffusion Scaling](https://mlsanity.com/paper/2603.02646/) (2026)
- [Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling](https://mlsanity.com/paper/2604.09159/) (2026)
- [CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making](https://mlsanity.com/paper/2605.01457/) (2026)
- [Refining Compositional Diffusion for Reliable Long-Horizon Planning](https://mlsanity.com/paper/2605.03075/) (2026)
- [HDFlow: Hierarchical Diffusion-Flow Planning for Long-horizon Tasks](https://mlsanity.com/paper/2605.04525/) (2026)
- [Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning](https://mlsanity.com/paper/2606.10613/) (2026)
- [Actor-Critic without Actor](https://mlsanity.com/paper/2509.21022/) (2025)
- [DADP: Domain Adaptive Diffusion Policy](https://mlsanity.com/paper/2602.04037/) (2026)
- [Direct Soft-Policy Sampling via Langevin Dynamics](https://mlsanity.com/paper/2602.07873/) (2026)
- [Improving Diffusion Planners by Self-Supervised Action Gating with Energies](https://mlsanity.com/paper/2603.02650/) (2026)
- [RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidance](https://mlsanity.com/paper/2606.27766/) (2026)
- [What Do You Need for Compositional Generalization in Diffusion Planning?](https://mlsanity.com/paper/2505.18083/) (2025)
- [Fast and Stable Diffusion Planning through Variational Adaptive Weighting](https://mlsanity.com/paper/2506.16688/) (2025)
- [Mixed-Density Diffuser: Efficient Planning with Non-Uniform Temporal Resolution](https://mlsanity.com/paper/2510.23026/) (2025)

## Abstract (arXiv)

Diffusion models have recently shown significant potential in solving decision-making problems, particularly in generating behavior plans -- also known as diffusion planning. While numerous studies have demonstrated the impressive performance of diffusion planning, the mechanisms behind the key components of a good diffusion planner remain unclear and the design choices are highly inconsistent in existing studies. In this work, we address this issue through systematic empirical experiments on diffusion planning in an offline reinforcement learning (RL) setting, providing practical insights into the essential components of diffusion planning. We trained and evaluated over 6,000 diffusion models, identifying the critical components such as guided sampling, network architecture, action generation and planning strategy. We revealed that some design choices opposite to the common practice in previous work in diffusion planning actually lead to better performance, e.g., unconditional sampling with selection can be better than guided sampling and Transformer outperforms U-Net as denoising network. Based on these insights, we suggest a simple yet strong diffusion planning baseline that achieves state-of-the-art results on standard offline RL benchmarks.