# Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning

arXiv: https://arxiv.org/abs/2305.18459 · HTML: https://mlsanity.com/paper/2305.18459/

## Key facts

- Cited by 115 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 73 of its 79 references that are in the map.
- First posted on arXiv 29 May 2023.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 73)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)

## Built on by (most cited first, 30 of 115)

- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [Diffusion Models for Reinforcement Learning: A Survey](https://mlsanity.com/paper/2311.01223/) (2023)
- [SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution](https://mlsanity.com/paper/2312.11598/) (2023)
- [RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning](https://mlsanity.com/paper/2510.14830/) (2025)
- [ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulation](https://mlsanity.com/paper/2406.01586/) (2024)
- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skills](https://mlsanity.com/paper/2506.12851/) (2025)
- [DiffuserLite: Towards Real-time Diffusion Planning](https://mlsanity.com/paper/2401.15443/) (2024)
- [Policy-Guided Diffusion](https://mlsanity.com/paper/2404.06356/) (2024)
- [Generative Trajectory Stitching through Diffusion Composition](https://mlsanity.com/paper/2503.05153/) (2025)
- [World Models via Policy-Guided Trajectory Diffusion](https://mlsanity.com/paper/2312.08533/) (2023)
- [Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training](https://mlsanity.com/paper/2402.14407/) (2024)
- [GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.16907/) (2024)
- [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024)
- [SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation](https://mlsanity.com/paper/2405.19586/) (2024)
- [LDP: A Local Diffusion Planner for Efficient Robot Navigation and Collision Avoidance](https://mlsanity.com/paper/2407.01950/) (2024)
- [DiffPhyCon: A Generative Approach to Control Complex Physical Systems](https://mlsanity.com/paper/2407.06494/) (2024)
- [Diffusion Model for Planning: A Systematic Literature Review](https://mlsanity.com/paper/2408.10266/) (2024)
- [Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement](https://mlsanity.com/paper/2410.11448/) (2024)
- [Variational Distillation of Diffusion Policies into Mixture of Experts](https://mlsanity.com/paper/2406.12538/) (2024)
- [Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimization](https://mlsanity.com/paper/2407.01624/) (2024)
- [Generative Models in Decision Making: A Survey](https://mlsanity.com/paper/2502.17100/) (2025)
- [Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer](https://mlsanity.com/paper/2505.24378/) (2025)
- [HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.18080/) (2024)
- [Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining](https://mlsanity.com/paper/2410.00564/) (2024)
- [Resisting Stochastic Risks in Diffusion Planners with the Trajectory Aggregation Tree](https://mlsanity.com/paper/2405.17879/) (2024)
- [Prioritized Generative Replay](https://mlsanity.com/paper/2410.18082/) (2024)
- [Diffusion Models as Network Optimizers: Explorations and Analysis](https://mlsanity.com/paper/2411.00453/) (2024)
- [State-Covering Trajectory Stitching for Diffusion Planners](https://mlsanity.com/paper/2506.00895/) (2025)
- [Efficient Planning with Latent Diffusion](https://mlsanity.com/paper/2310.00311/) (2023)

## Abstract (arXiv)

Diffusion models have demonstrated highly-expressive generative capabilities in vision and NLP. Recent studies in reinforcement learning (RL) have shown that diffusion models are also powerful in modeling complex policies or trajectories in offline datasets. However, these works have been limited to single-task settings where a generalist agent capable of addressing multi-task predicaments is absent. In this paper, we aim to investigate the effectiveness of a single diffusion model in modeling large-scale multi-task offline data, which can be challenging due to diverse and multimodal data distribution. Specifically, we propose Multi-Task Diffusion Model (textscMTDiff), a diffusion-based method that incorporates Transformer backbones and prompt learning for generative planning and data synthesis in multi-task offline settings. textscMTDiff leverages vast amounts of knowledge available in multi-task data and performs implicit knowledge sharing among tasks. For generative planning, we find textscMTDiff outperforms state-of-the-art algorithms across 50 tasks on Meta-World and 8 maps on Maze2D. For data synthesis, textscMTDiff generates high-quality data for testing tasks given a single demonstration as a prompt, which enhances the low-quality datasets for even unseen tasks.