mlsanityData to 14 Oct 2026

Diffusion Models for Reinforcement Learning: A Survey

arXiv 2311.01223 ↗First posted 2 Nov 2023cs.LG cs.AIcited by 91

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Diffusion models surpass previous generative models in sample quality and training stability. Recent works have shown the advantages of diffusion models in improving reinforcement learning (RL) solutions. This survey aims to provide an overview of this emerging field and hopes to inspire new avenues of research. First, we examine several challenges encountered by RL algorithms. Then, we present a taxonomy of existing methods based on the roles of diffusion models in RL and explore how the preceding challenges are addressed. We further outline successful applications of diffusion models in various RL-related tasks. Finally, we conclude the survey and offer insights into future research directions. We are actively maintaining a GitHub repository for papers and other related resources in utilizing diffusion models in RL: https://github.com/apexrl/Diff4RLSurvey.

Key facts. Cited by 91 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 113 of its 136 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

113 of 136 references.

Closest ancestors · 30

Same direction, newest first

  1. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models16 Oct
  2. 2023Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models13 Oct
  3. 2023Boosting Continuous Control with Consistency Policy10 Oct
  4. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning9 Oct
  5. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model3 Oct
  6. 2023Efficient Planning with Latent Diffusion30 Sep
  7. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning12 Sep
  8. 2023EquiDiff: A Conditional Equivariant Diffusion Model For Trajectory Prediction12 Aug
  9. 2023Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models3 Aug
  10. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning10 Jul
  11. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learning4 Jul
  12. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RL4 Jul
18 more
  1. 2023Value function estimation using conditional diffusion models for control9 Jun
  2. 2023Extracting Reward Functions from Diffusion Models1 Jun
  3. 2023SafeDiffuser: Safe Planning with Diffusion Probabilistic Models31 May
  4. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  5. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL31 May
  6. 2023Generating Behaviorally Diverse Policies with Latent Diffusion Models30 May
  7. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning29 May
  8. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  9. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  10. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  11. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  12. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  13. 2023To the Noise and Back: Diffusion for Shared Autonomy23 Feb
  14. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  15. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  16. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  17. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics5 Oct
  18. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 83

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  10. 2013Auto-Encoding Variational Bayescited by 4,053
  11. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
38 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  9. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  11. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  12. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  13. 2017Constrained Policy Optimizationcited by 983
  14. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  15. 2022A Generalist Agentcited by 808
  16. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
  17. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 735
  18. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  19. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  20. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
  21. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  22. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 516
  23. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  24. 2020Reinforcement Learning with Augmented Datacited by 468
  25. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  26. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  27. 2018Reward Constrained Policy Optimizationcited by 417
  28. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
  29. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 297
  30. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 287
  31. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  32. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  33. 2022Multi-Game Decision Transformerscited by 214
  34. 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusioncited by 213
  35. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  36. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  37. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  38. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 31; 2025: 34; 2026: 23.

91 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 280
  2. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 110
  3. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 98
  4. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 73
  5. 2024DreamReward: Text-to-3D Generation with Human Preferencecited by 56
  6. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  7. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 42
  8. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 39
  9. 2025What Makes a Good Diffusion Planner for Decision Making?cited by 38
  10. 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
  11. 2024Attacks and Defenses for Generative Diffusion Models: A Comprehensive Surveycited by 32
  12. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learningcited by 31
38 more
  1. 2024Flow Matching Imitation Learning for Multi-Support Manipulationcited by 29
  2. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 28
  3. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 27
  4. 2023Diffusion Illusions: Hiding Images in Plain Sightcited by 26
  5. 2025Prior Does Matter: Visual Navigation via Denoising Diffusion Bridge Modelscited by 25
  6. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  7. 2024LDP: A Local Diffusion Planner for Efficient Robot Navigation and Collision Avoidancecited by 18
  8. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  9. 2025Generative Models in Decision Making: A Surveycited by 15
  10. 2025Wasserstein Convergence of Score-based Generative Models under Semiconvexity and Discontinuous Gradientscited by 15
  11. 2024Prioritized Generative Replaycited by 13
  12. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
  13. 2025Implicit Search via Discrete Diffusion: A Study on Chesscited by 11
  14. 2025Closing the Train-Test Gap in World Models for Gradient-Based Planningcited by 11
  15. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  16. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  17. 2025DiffRefiner: Coarse to Fine Trajectory Planning via Diffusion Refinement with Semantic Interaction for End to End Autonomous Drivingcited by 10
  18. 2025Latent Theory of Mind: A Decentralized Diffusion Architecture for Cooperative Manipulationcited by 9
  19. 2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9
  20. 2024Diffusion-Reinforcement Learning Hierarchical Motion Planning in Multi-agent Adversarial Gamescited by 7
  21. 2024Preference Aligned Diffusion Planner for Quadrupedal Locomotion Controlcited by 7
  22. 2024Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replaycited by 7
  23. 2025Diffusion Models for Smarter UAVs: Decision-Making and Modelingcited by 7
  24. 2025CHD: Coupled Hierarchical Diffusion for Long-Horizon Taskscited by 6
  25. 2024Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Datacited by 5
  26. 2025Exploratory Diffusion Model for Unsupervised Reinforcement Learningcited by 5
  27. 2023iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Designcited by 4
  28. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  29. 2024MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learningcited by 4
  30. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  31. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 3
  32. 2024Neural Shr"odinger Bridge Matching for Pansharpeningcited by 2
  33. 2024DyDiff: Long-Horizon Rollout via Dynamics Diffusion for Offline Reinforcement Learningcited by 2
  34. 2024Reinforcement Learning for Sustainable Energy: A Surveycited by 2
  35. 2024Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsalcited by 2
  36. 2025Extendable Planning via Multiscale Diffusioncited by 2
  37. 2025Improved Sample Complexity For Diffusion Model Training Without Empirical Risk Minimizer Accesscited by 2
  38. 2025STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluationcited by 2

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View
  2. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  3. 2023Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization
  4. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning
  5. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design
  6. 2025Connections between reinforcement learning with feedback,test-time scaling, and diffusion guidance: An anthology
  7. 2024Diffusion Spectral Representation for Reinforcement Learning
  8. 2024Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective