mlsanityData to 14 Oct 2026

Efficient Diffusion Policies for Offline Reinforcement Learning

arXiv 2305.20081 ↗First posted 31 May 2023cs.LG cs.AIcited by 157

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Offline reinforcement learning (RL) aims to learn optimal policies from offline datasets, where the parameterization of policies is crucial but often overlooked. Recently, Diffsuion-QL significantly boosts the performance of offline RL by representing a policy with a diffusion model, whose success relies on a parametrized Markov Chain with hundreds of steps for sampling. However, Diffusion-QL suffers from two critical limitations. 1) It is computationally inefficient to forward and backward through the whole Markov chain during training. 2) It is incompatible with maximum likelihood-based RL algorithms (e.g., policy gradient methods) as the likelihood of diffusion models is intractable. Therefore, we propose efficient diffusion policy (EDP) to overcome these two challenges. EDP approximately constructs actions from corrupted ones at training to avoid running the sampling chain. We conduct extensive experiments on the D4RL benchmark. The results show that EDP can reduce the diffusion policy training time from 5 days to 5 hours on gym-locomotion tasks. Moreover, we show that EDP is compatible with various offline RL algorithms (TD3, CRR, and IQL) and achieves new state-of-the-art on D4RL by large margins over previous methods. Our code is available at https://github.com/sail-sg/edp.

Key facts. Cited by 157 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 33 of its 37 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

33 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  9. 2015Continuous control with deep reinforcement learningcited by 3,407
  10. 2015Trust Region Policy Optimizationcited by 2,580
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
20 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  13. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  14. 2020Critic Regularized Regressioncited by 275
  15. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  16. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  17. 2021Offline RL Without Off-Policy Evaluationcited by 151
  18. 2019Mish: A Self Regularized Non-Monotonic Activation Functioncited by 141
  19. 2022Boosting Offline Reinforcement Learning via Data Rebalancingcited by 22
  20. 2022Mutual Information Regularized Offline Reinforcement Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 33; 2025: 54; 2026: 63.

157 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 280
  2. 2025Flow Q-Learningcited by 184
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  4. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
  5. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 110
  6. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 94
  7. 2024Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulationcited by 93
  8. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  9. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 73
  10. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 72
  11. 2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 71
  12. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
38 more
  1. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 59
  2. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 58
  3. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 58
  4. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  5. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  6. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learningcited by 43
  7. 2024Potential Based Diffusion Motion Planningcited by 41
  8. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  9. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 39
  10. 2023Boosting Continuous Control with Consistency Policycited by 38
  11. 2025Flow-Based Policy for Online Reinforcement Learningcited by 37
  12. 2026Q-learning with Adjoint Matchingcited by 37
  13. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 34
  14. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 34
  15. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
  16. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  17. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learningcited by 31
  18. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
  19. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 28
  20. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  21. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  22. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  23. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 20
  24. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 20
  25. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
  26. 2024Sampling from Energy-based Policies using Diffusioncited by 17
  27. 2026Flow Policy Gradients for Robot Controlcited by 16
  28. 2023Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RLcited by 15
  29. 2024Variational Distillation of Diffusion Policies into Mixture of Expertscited by 15
  30. 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learningcited by 15
  31. 2025Enhanced DACER Algorithm with High Diffusion Efficiencycited by 13
  32. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Controlcited by 12
  33. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
  34. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learningcited by 11
  35. 2026Unleashing the Potential of Diffusion Models for End-to-End Autonomous Drivingcited by 11
  36. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 10
  37. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
  38. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 9

50 most cited of 157 All 157 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RL
  2. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching
  3. 2025DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
  4. 2025Offline Reinforcement Learning with Discrete Diffusion Skills
  5. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  6. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning
  7. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Models
  8. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond