Data to 7 Oct 2026

Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning

arXiv 2305.18459 ↗First posted 29 May 2023cs.LG cs.AIcited by 114

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion models have demonstrated highly-expressive generative capabilities in vision and NLP. Recent studies in reinforcement learning (RL) have shown that diffusion models are also powerful in modeling complex policies or trajectories in offline datasets. However, these works have been limited to single-task settings where a generalist agent capable of addressing multi-task predicaments is absent. In this paper, we aim to investigate the effectiveness of a single diffusion model in modeling large-scale multi-task offline data, which can be challenging due to diverse and multimodal data distribution. Specifically, we propose Multi-Task Diffusion Model (textscMTDiff), a diffusion-based method that incorporates Transformer backbones and prompt learning for generative planning and data synthesis in multi-task offline settings. textscMTDiff leverages vast amounts of knowledge available in multi-task data and performs implicit knowledge sharing among tasks. For generative planning, we find textscMTDiff outperforms state-of-the-art algorithms across 50 tasks on Meta-World and 8 maps on Maze2D. For data synthesis, textscMTDiff generates high-quality data for testing tasks given a single demonstration as a prompt, which enhances the low-quality datasets for even unseen tasks.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 73 of its 79 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

73 of 79 references.

Closest ancestors · 5

Same direction, newest first

Other references · 68

2017Attention Is All You Needcited by 59,932
2014Adam: A Method for Stochastic Optimizationcited by 36,960
2020Language Models are Few-Shot Learnerscited by 32,199
2020Denoising Diffusion Probabilistic Modelscited by 20,851
2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
2022Scalable Diffusion Models with Transformerscited by 5,931
2022Classifier-Free Diffusion Guidancecited by 5,717
2016Layer Normalizationcited by 5,579

38 more

2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2020Gradient Surgery for Multi-Task Learningcited by 1,210
2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2022A Generalist Agentcited by 802
2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
2020Reinforcement Learning with Augmented Datacited by 452
2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 441
2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 439
2018Quantifying Generalization in Reinforcement Learningcited by 345
2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
2022Multi-Game Decision Transformerscited by 212
2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 51; 2025: 41; 2026: 17.

114 papers.

Sort:

By area:

20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 69
2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
2024DiffuserLite: Towards Real-time Diffusion Planningcited by 40
2024Policy-Guided Diffusioncited by 40
2025KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skillscited by 39
2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
2023World Models via Policy-Guided Trajectory Diffusioncited by 34
2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26

38 more

2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
2024LDP: A Local Diffusion Planner for Efficient Robot Navigation and Collision Avoidancecited by 18
2024DiffPhyCon: A Generative Approach to Control Complex Physical Systemscited by 17
2024Diffusion Model for Planning: A Systematic Literature Reviewcited by 17
2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
2024Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimizationcited by 15
2025Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformercited by 15
2024HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learningcited by 14
2024Variational Distillation of Diffusion Policies into Mixture of Expertscited by 14
2025Generative Models in Decision Making: A Surveycited by 14
2024Resisting Stochastic Risks in Diffusion Planners with the Trajectory Aggregation Treecited by 13
2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretrainingcited by 13
2024Prioritized Generative Replaycited by 13
2024Diffusion Models as Network Optimizers: Explorations and Analysiscited by 13
2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
2023Efficient Planning with Latent Diffusioncited by 11
2024DDM-Lag : A Diffusion-based Decision-making Model for Autonomous Vehicles with Lagrangian Safety Enhancementcited by 11
2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 11
2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
2025Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspectivecited by 9
2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Modelscited by 8
2025Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperationcited by 8
2025Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planningcited by 8
2024Diffusion-Reinforcement Learning Hierarchical Motion Planning in Multi-agent Adversarial Gamescited by 7
2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
2024Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Surveycited by 6
2024Solving Continual Offline RL through Selective Weights Activation on Aligned Spacescited by 6
2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalizationcited by 6
2024Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replaycited by 6
2024GDSG: Graph Diffusion-based Solution Generator for Optimization Problems in MEC Networkscited by 6
2025CHD: Coupled Hierarchical Diffusion for Long-Horizon Taskscited by 6

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way