mlsanityData to 14 Oct 2026

Efficient Online Reinforcement Learning for Diffusion Policy

arXiv 2502.00361 ↗First posted 1 Feb 2025cs.LGcited by 71

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 2 more
Abstract (arXiv)

Diffusion policies have achieved superior performance in imitation learning and offline reinforcement learning (RL) due to their rich expressiveness. However, the conventional diffusion training procedure requires samples from target distribution, which is impossible in online RL since we cannot sample from the optimal policy. Backpropagating policy gradient through the diffusion process incurs huge computational costs and instability, thus being expensive and not scalable. To enable efficient training of diffusion policies in online RL, we generalize the conventional denoising score matching by reweighting the loss function. The resulting Reweighted Score Matching (RSM) preserves the optimal solution and low computational cost of denoising score matching, while eliminating the need to sample from the target distribution and allowing learning to optimize value functions. We introduce two tractable reweighted loss functions to solve two commonly used policy optimization problems, policy mirror descent and max-entropy policy, resulting in two practical algorithms named Diffusion Policy Mirror Descent (DPMD) and Soft Diffusion Actor-Critic (SDAC). We conducted comprehensive comparisons on MuJoCo benchmarks. The empirical results show that the proposed algorithms outperform recent diffusion-policy online RLs on most tasks, and the DPMD improves more than 120% over soft actor-critic on Humanoid and Ant.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 39 of its 51 references that are in the map. First posted on arXiv 1 Feb 2025.

Builds on

39 of 51 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Diffusion Policy Policy Optimization1 Sep
  2. 2024Diffusion Spectral Representation for Reinforcement Learning23 Jun
  3. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  4. 2024Diffusion Actor-Critic with Entropy Regulator24 May
  5. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representations16 Feb
  6. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  7. 2023World Models via Policy-Guided Trajectory Diffusion13 Dec
  8. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  9. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  10. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 28

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  11. 2015Trust Region Policy Optimizationcited by 2,580
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
16 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  3. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  4. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
  5. 2021How to Train Your Energy-Based Modelscited by 274
  6. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  7. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  8. 2017A unified view of entropy-regularized Markov decision processescited by 182
  9. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  10. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  11. 2022Flow Annealed Importance Sampling Bootstrapcited by 127
  12. 2024Iterated Denoising Energy Matching for Sampling from Boltzmann Densitiescited by 116
  13. 2020Mirror Descent Policy Optimizationcited by 106
  14. 2024DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasetscited by 49
  15. 2023Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objectscited by 40
  16. 2024Sampling from Energy-based Policies using Diffusioncited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 53.

71 papers.

Sort:

By area:

  1. 2026Q-learning with Adjoint Matchingcited by 37
  2. 2025One-Step Flow Policy Mirror Descentcited by 21
  3. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  4. 2025Enhanced DACER Algorithm with High Diffusion Efficiencycited by 13
  5. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policiescited by 12
  6. 2025Multi-agent Coordination via Flow Matchingcited by 10
  7. 2025Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspectivecited by 9
  8. 2025Relative Entropy Pathwise Policy Optimizationcited by 7
  9. 2025Dichotomous Diffusion Policy Optimizationcited by 6
  10. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  11. 2025Exploratory Diffusion Model for Unsupervised Reinforcement Learningcited by 5
  12. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
38 more
  1. 2026Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flowcited by 5
  2. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 5
  3. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  4. 2025Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraintscited by 4
  5. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 4
  6. 2026Reparameterization Flow Policy Optimizationcited by 4
  7. 2025RODS: Robust Optimization Inspired Diffusion Sampling for Detecting and Reducing Hallucination in Generative Modelscited by 3
  8. 2026Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policiescited by 3
  9. 2026GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policiescited by 3
  10. 2026FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policiescited by 3
  11. 2026Mean Flow Policy Optimizationcited by 3
  12. 2026FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidancecited by 3
  13. 2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Controlcited by 2
  14. 2026RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedbackcited by 2
  15. 2026What Does Flow Matching Bring To TD Learning?cited by 2
  16. 2026Diffusion Controller: Framework, Algorithms and Parameterizationcited by 2
  17. 2026REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learningcited by 2
  18. 2026Flow Matching Policy Optimization with Mirror Descent and Entropy Constraintscited by 2
  19. 2026Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Samplingcited by 2
  20. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 2
  21. 2026Score-Based One-step MeanFlow Policy Optimizationcited by 2
  22. 2026Reinforcement Learning for Flow-Matching Policies with Density Transportcited by 2
  23. 2026Learning When to Stop: Prefix-Optimal Dynamic Diffusion Policies for Continuous Controlcited by 2
  24. 2025Actor-Critic without Actorcited by 1
  25. 2025Accelerating Inference of Masked Image Generators via Reinforcement Learningcited by 1
  26. 2025Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQRcited by 1
  27. 2026Latent Spherical Flow Policy for Reinforcement Learning with Combinatorial Actionscited by 1
  28. 2026Direct Soft-Policy Sampling via Langevin Dynamicscited by 1
  29. 2026Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Drivingcited by 1
  30. 2026Online Self-Training for Co-Adaptation in Hierarchical Diffusion Policiescited by 1
  31. 2026GeMPO: Generalized Measure Matching for Online Diffusion Reinforcement Learningcited by 1
  32. 2026Advantage-Guided Diffusion for Model-Based Reinforcement Learningcited by 1
  33. 2026Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learningcited by 1
  34. 2026Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with k-step Policy Gradientscited by 1
  35. 2026BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Modelscited by 1
  36. 2026X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matchingcited by 1
  37. 2025Act to See, See to Act: Diffusion-Driven Perception-Action Interplay for Adaptive Policiescited by 0
  38. 2025RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularizationcited by 0

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning
  2. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
  3. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  4. 2023Score Regularized Policy Optimization through Diffusion Behavior
  5. 2026Reinforcement Learning from Denoising Feedback
  6. 2025Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transport
  7. 2023Efficient Diffusion Policies for Offline Reinforcement Learning
  8. 2025Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models