mlsanityData to 14 Oct 2026

Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control

arXiv 2510.17122 ↗First posted 20 Oct 2025cs.LG math.OCcited by 4

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement learning (RL) has achieved significant success across a wide range of domains, however, most existing methods are formulated in discrete time. In this work, we introduce a novel RL method for continuous-time control, where stochastic differential equations govern state-action dynamics. Departing from traditional value function-based approaches, our key contribution is the characterization of continuous-time Q-functions via a martingale condition and the linking of diffusion policy scores to the action gradient of a learned continuous Q-function by the dynamic programming principle. This insight motivates Continuous Q-Score Matching (CQSM), a score-based policy improvement algorithm. Notably, our method addresses a long-standing challenge in continuous-time RL: preserving the action-evaluation capability of Q-functions without relying on time discretization. We further provide theoretical closed-form solutions for linear-quadratic (LQ) control problems within our framework. Numerical results in simulated environments demonstrate the effectiveness of our proposed method and compare it to popular baselines.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 39 of its 59 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

39 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning29 Jan
  2. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes16 Jan

Other references · 37

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  12. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
25 more
  1. 2023Imitating Human Behaviour with Diffusion Modelscited by 263
  2. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 236
  3. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 233
  4. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  5. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
  6. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 110
  7. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  8. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 94
  9. 2022q-Learning in Continuous Timecited by 91
  10. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  11. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 71
  12. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
  13. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  14. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  15. 2021Exploratory HJB equations and their convergencecited by 32
  16. 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 31
  17. 2023Policy Optimization for Continuous Reinforcement Learningcited by 31
  18. 2021Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controlscited by 24
  19. 2022Solving Continuous Control via Q-learningcited by 24
  20. 2020Learning to Represent Action Values as a Hypergraph on the Action Verticescited by 23
  21. 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 21
  22. 2020State-Dependent Temperature Control for Langevin Diffusionscited by 20
  23. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 19
  24. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 19
  25. 2016Linear Quadratic Mean Field Type Control and Mean Field Games with Common Noise, with Application to Production of an Exhaustible Resourcecited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policiescited by 3
  2. 2026Path Integral Value Matching for Linear Quadratic Stochastic Optimal Controlcited by 0
  3. 2026iADD: Improving Alignment and Diversity in Diffusion Policy Optimizationcited by 0
  4. 2026Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning
  2. 2023Boosting Continuous Control with Consistency Policy
  3. 2025Policy Transfer for Continuous-Time Reinforcement Learning: A (Rough) Differential Equation Approach
  4. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  5. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models
  6. 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation
  7. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control
  8. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning