mlsanityData to 7 Oct 2026

Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning

arXiv 2208.06193 ↗First posted 12 Aug 2022cs.LG stat.MLcited by 543

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Offline reinforcement learning (RL), which aims to learn an optimal policy using a previously collected static dataset, is an important paradigm of RL. Standard RL methods often perform poorly in this regime due to the function approximation errors on out-of-distribution actions. While a variety of regularization methods have been proposed to mitigate this issue, they are often constrained by policy classes with limited expressiveness that can lead to highly suboptimal solutions. In this paper, we propose representing the policy as a diffusion model, a recent class of highly-expressive deep generative models. We introduce Diffusion Q-learning (Diffusion-QL) that utilizes a conditional diffusion model to represent the policy. In our approach, we learn an action-value function and we add a term maximizing action-values into the training loss of the conditional diffusion model, which results in a loss that seeks optimal actions that are near the behavior policy. We show the expressiveness of the diffusion model-based policy, and the coupling of the behavior cloning and policy improvement under the diffusion model both contribute to the outstanding performance of Diffusion-QL. We illustrate the superiority of our method compared to prior works in a simple 2D bandit example with a multimodal behavior policy. We then show that our method can achieve state-of-the-art performance on the majority of the D4RL benchmark tasks.

Key facts. Cited by 543 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 31 of its 45 references that are in the map. First posted on arXiv 12 Aug 2022.

Builds on

31 of 45 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  12. 2016Variational Inference: A Review for Statisticianscited by 1,466
19 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  13. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  14. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  15. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  16. 2022Diffusion-GAN: Training GANs with Diffusioncited by 203
  17. 2021Offline RL Without Off-Policy Evaluationcited by 150
  18. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  19. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 74; 2024: 129; 2025: 172; 2026: 160.

543 papers.

Sort:

By area:

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  2. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  3. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  4. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  5. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  6. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  7. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  8. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  9. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  10. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  11. 2024Diffusion Policy Policy Optimizationcited by 255
  12. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 234
38 more
  1. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  2. 2022Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptionscited by 215
  3. 2025Real-Time Execution of Action Chunking Flow Policiescited by 199
  4. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  5. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  6. 2025Flow Q-Learningcited by 162
  7. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  8. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  9. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  10. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  11. 2023The probability flow ODE is provably fastcited by 145
  12. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Modelscited by 131
  13. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
  14. 2022SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusioncited by 125
  15. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  16. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  17. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  18. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  19. 2022How to Backdoor Diffusion Models?cited by 102
  20. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  21. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  22. 2024FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulationcited by 99
  23. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  24. 2023Compositional Foundation Models for Hierarchical Planningcited by 92
  25. 2024Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulationcited by 92
  26. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  27. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  28. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 83
  29. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  30. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81
  31. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  32. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 73
  33. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  34. 2024Simple Hierarchical Planning with Diffusioncited by 71
  35. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  36. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 70
  37. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 70
  38. 2023VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Modelscited by 68

50 most cited of 543 All 543 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Generating Behaviorally Diverse Policies with Latent Diffusion Models
  2. 2024Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective
  3. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow
  4. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning
  5. 2026Expressivity and Statistical Trade-offs in Diffusion Policy Learning
  6. 2025DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
  7. 2024Adding Conditional Control to Diffusion Models with Reinforcement Learning
  8. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions