mlsanityData to 7 Oct 2026

Learning a Diffusion Model Policy from Rewards via Q-Score Matching

arXiv 2312.11752 ↗First posted 18 Dec 2023cs.LG cs.AIcited by 126

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Diffusion models have become a popular choice for representing actor policies in behavior cloning and offline reinforcement learning. This is due to their natural ability to optimize an expressive class of distributions over a continuous space. However, previous works fail to exploit the score-based structure of diffusion models, and instead utilize a simple behavior cloning term to train the actor, limiting their ability in the actor-critic setting. In this paper, we present a theoretical framework linking the structure of diffusion model policies to a learned Q-function, by linking the structure between the score of the policy to the action gradient of the Q-function. We focus on off-policy reinforcement learning and propose a new policy update method from this theory, which we denote Q-score matching. Notably, this algorithm only needs to differentiate through the denoising model rather than the entire diffusion model evaluation, and converged policies through Q-score matching are implicitly multi-modal and explorative in continuous domains. We conduct experiments in simulated environments to demonstrate the viability of our proposed method and compare to popular baselines. Source code is available from the project website: https://michaelpsenka.io/qsm.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 26 of its 51 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

26 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  2. 2023Training Diffusion Models with Reinforcement Learning22 May
  3. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  4. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  5. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 20

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  8. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  9. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  10. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  11. 2022Novel View Synthesis with Diffusion Modelscited by 299
  12. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
8 more
  1. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  2. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  3. 2022q-Learning in Continuous Timecited by 68
  4. 2021Recomposing the Reinforcement Learning Building Blocks with Hypernetworkscited by 33
  5. 2023Policy Optimization for Continuous Reinforcement Learningcited by 30
  6. 2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matchingcited by 17
  7. 2020How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimizationcited by 16
  8. 2021Exponential Family Model-Based Reinforcement Learning via Score Matchingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 39; 2026: 72.

126 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 255
  2. 2025Flow Q-Learningcited by 162
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  4. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  5. 2025Flow Matching Policy Gradientscited by 86
  6. 2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 83
  7. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  8. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81
  9. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  10. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
  11. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
  12. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
38 more
  1. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
  2. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  3. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  4. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  5. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  6. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  7. 2025Flow-Based Policy for Online Reinforcement Learningcited by 33
  8. 2026Q-learning with Adjoint Matchingcited by 33
  9. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  10. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  11. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 29
  12. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
  13. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
  14. 2025One-Step Flow Policy Mirror Descentcited by 18
  15. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 17
  16. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
  17. 2026Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Designcited by 17
  18. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 16
  19. 2024Sampling from Energy-based Policies using Diffusioncited by 15
  20. 2026Flow Policy Gradients for Robot Controlcited by 14
  21. 2025Enhanced DACER Algorithm with High Diffusion Efficiencycited by 13
  22. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
  23. 2025D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learningcited by 10
  24. 2025SuperFlow: Training Flow Matching Models with RL on the Flycited by 10
  25. 2025Unleashing Flow Policies with Distributional Criticscited by 7
  26. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policiescited by 7
  27. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 7
  28. 2025Habitizing Diffusion Planning for Efficient and Effective Decision Makingcited by 6
  29. 2025Uncertainty Comes for Free: Human-in-the-Loop Policies with Diffusion Modelscited by 6
  30. 2025CHD: Coupled Hierarchical Diffusion for Long-Horizon Taskscited by 6
  31. 2025Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timestepscited by 6
  32. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  33. 2025Quantization-Free Autoregressive Action Transformercited by 5
  34. 2025NeuralRemaster: Phase-Preserving Diffusion for Structure-Aligned Generationcited by 5
  35. 2026PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learningcited by 5
  36. 2026FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matchingcited by 5
  37. 2026You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vectorcited by 5
  38. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 5

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward-Directed Score-Based Diffusion Models via q-Learning
  2. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning
  3. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  4. 2023Score Regularized Policy Optimization through Diffusion Behavior
  5. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning
  6. 2019Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments
  7. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning
  8. 2023Learning to Reach Goals via Diffusion