mlsanityData to 7 Oct 2026

Policy Representation via Diffusion Probability Model for Reinforcement Learning

arXiv 2305.13122 ↗First posted 22 May 2023cs.LGcited by 105

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Popular reinforcement learning (RL) algorithms tend to produce a unimodal policy distribution, which weakens the expressiveness of complicated policy and decays the ability of exploration. The diffusion probability model is powerful to learn complicated multimodal distributions, which has shown promising and potential applications to RL. In this paper, we formally build a theoretical foundation of policy representation via the diffusion probability model and provide practical implementations of diffusion policy for online model-free RL. Concretely, we character diffusion policy as a stochastic process, which is a new approach to representing a policy. Then we present a convergence guarantee for diffusion policy, which provides a theory to understand the multimodality of diffusion policy. Furthermore, we propose the DIPO which is an implementation for model-free online RL with DIffusion POlicy. To the best of our knowledge, DIPO is the first algorithm to solve model-free online RL problems with the diffusion model. Finally, extensive empirical results show the effectiveness and superiority of DIPO on the standard continuous control Mujoco benchmark.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 60 of its 98 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

60 of 98 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  2. 2023EDGI: Equivariant Diffusion for Planning with Embodied Agents22 Mar
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  4. 2023CFlowNets: Continuous Control with Generative Flow Networks4 Mar
  5. 2023ReorientDiff: Diffusion Model based Reorientation for Object Manipulation28 Feb
  6. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  7. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  8. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  9. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics5 Oct
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2015Variational Inference with Normalizing Flowscited by 2,161
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
38 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  6. 2022A Generalist Agentcited by 802
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 526
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  10. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  11. 2021Implicit Behavioral Cloningcited by 442
  12. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptionscited by 440
  13. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  14. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  15. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  16. 2019Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Sufficescited by 224
  17. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  18. 2022Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptionscited by 215
  19. 2022Online Decision Transformercited by 194
  20. 2022Convergence of score-based generative modeling for general data distributionscited by 179
  21. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  22. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  23. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  24. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  25. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  26. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  27. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  28. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  29. 2021Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimationcited by 95
  30. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  31. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  32. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 70
  33. 2018Sampling Can Be Faster Than Optimizationcited by 68
  34. 2020Energy-Based Imitation Learningcited by 39
  35. 2023Stochastic Generative Flow Networkscited by 32
  36. 2021Hierarchical Policies for Cluttered-Scene Grasping with Latent Planscited by 12
  37. 2022ARC - Actor Residual Critic for Adversarial Imitation Learningcited by 3
  38. 2023Diverse Policy Optimization for Structured Action Spacecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 16; 2025: 43; 2026: 42.

105 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 255
  2. 2025Flow Q-Learningcited by 162
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  4. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  5. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  6. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81
  7. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  8. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  9. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 69
  10. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
  11. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  12. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 59
38 more
  1. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
  2. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  3. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  4. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  5. 2023Boosting Continuous Control with Consistency Policycited by 36
  6. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  7. 2025Flow-Based Policy for Online Reinforcement Learningcited by 33
  8. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  9. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  10. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 29
  11. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
  12. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 26
  13. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  14. 2025UniDB: A Unified Diffusion Bridge Framework via Stochastic Optimal Controlcited by 18
  15. 2025One-Step Flow Policy Mirror Descentcited by 18
  16. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 17
  17. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
  18. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 16
  19. 2024Sampling from Energy-based Policies using Diffusioncited by 15
  20. 2025Generative Models in Decision Making: A Surveycited by 14
  21. 2025Falcon: Fast Visuomotor Policies via Partial Denoisingcited by 14
  22. 2025Responsive Noise-Relaying Diffusion Policy: Responsive and Efficient Visuomotor Controlcited by 13
  23. 2025Enhanced DACER Algorithm with High Diffusion Efficiencycited by 13
  24. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
  25. 2025D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learningcited by 10
  26. 2025Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspectivecited by 9
  27. 2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9
  28. 2025Spatial-Temporal Aware Visuomotor Diffusion Policy Learningcited by 8
  29. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 8
  30. 2025Unleashing Flow Policies with Distributional Criticscited by 7
  31. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 7
  32. 2025Habitizing Diffusion Planning for Efficient and Effective Decision Makingcited by 6
  33. 2025Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transportcited by 6
  34. 2025Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timestepscited by 6
  35. 2025IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuningcited by 5
  36. 2026FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matchingcited by 5
  37. 2026You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vectorcited by 5
  38. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 5

50 most cited of 105 All 105 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  2. 2025Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
  3. 2024Diffusion Spectral Representation for Reinforcement Learning
  4. 2025Dichotomous Diffusion Policy Optimization
  5. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?
  6. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  7. 2023Score Regularized Policy Optimization through Diffusion Behavior
  8. 2026Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View