mlsanityData to 7 Oct 2026

q-Learning in Continuous Time

arXiv 2207.00713 ↗First posted 2 Jul 2022cs.LG cs.AI q-fin.CPcited by 68

Not on the map. Why?

Abstract (arXiv)

We study the continuous-time counterpart of Q-learning for reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation introduced by Wang et al. (2020). As the conventional (big) Q-function collapses in continuous time, we consider its first-order approximation and coin the term ``(little) q-function". This function is related to the instantaneous advantage rate function as well as the Hamiltonian. We develop a ``q-learning" theory around the q-function that is independent of time discretization. Given a stochastic policy, we jointly characterize the associated q-function and value function by martingale conditions of certain stochastic processes, in both on-policy and off-policy settings. We then apply the theory to devise different actor-critic algorithms for solving underlying RL problems, depending on whether or not the density function of the Gibbs measure generated from the q-function can be computed explicitly. One of our algorithms interprets the well-known Q-learning algorithm SARSA, and another recovers a policy gradient (PG) based continuous-time algorithm proposed in Jia and Zhou (2022b). Finally, we conduct simulation experiments to compare the performance of our algorithms with those of PG-based algorithms in Jia and Zhou (2022b) and time-discretized conventional Q-learning algorithms.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 20 of its 48 references that are in the map. First posted on arXiv 2 Jul 2022.

Builds on

20 of 48 references.

References · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  6. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  7. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  8. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  9. 2017Deep Exploration via Randomized Value Functionscited by 155
  10. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  11. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  12. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
8 more
  1. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
  2. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
  3. 2019Making Deep Q-learning methods robust to time discretizationcited by 51
  4. 2018Q-learning with Nearest Neighborscited by 37
  5. 2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
  6. 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 29
  7. 2021Exploratory HJB equations and their convergencecited by 28
  8. 2020State-Dependent Temperature Control for Langevin Diffusionscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 21; 2025: 17; 2026: 19.

68 papers.

Sort:

By area:

  1. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
  2. 2024Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorialcited by 60
  3. 2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
  4. 2023Policy Optimization for Continuous Reinforcement Learningcited by 30
  5. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 19
  6. 2023Actor-Critic learning for mean-field control in continuous timecited by 19
  7. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learningcited by 19
  8. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 18
  9. 2023Continuous-time q-learning for mean-field control problemscited by 17
  10. 2023Fast Policy Learning for Linear Quadratic Control with Entropy Regularizationcited by 11
  11. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problemscited by 10
  12. 2024Regret of exploratory policy improvement and q-learningcited by 10
38 more
  1. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 9
  2. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learningcited by 9
  3. 2024Control randomisation approach for policy gradient and application to reinforcement learning in optimal switchingcited by 7
  4. 2024Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropycited by 7
  5. 2024Unified continuous-time q-learning for mean-field game and mean-field control problemscited by 7
  6. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  7. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and Statecited by 7
  8. 2023Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spacescited by 6
  9. 2023Data-Driven Merton's Strategies via Policy Randomizationcited by 6
  10. 2025Deep Learning for Continuous-Time Stochastic Control with Jumpscited by 6
  11. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 5
  12. 2024Exploratory Optimal Stopping: A Singular Control Formulationcited by 5
  13. 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundationscited by 5
  14. 2024On Bellman equations for continuous-time policy evaluation I: discretization and approximationcited by 4
  15. 2025Continuous-time reinforcement learning for optimal switching over multiple regimescited by 4
  16. 2022Square-root regret bounds for continuous-time episodic Markov decision processescited by 3
  17. 2024Reinforcement Learning for Intensity Control: An Application to Choice-Based Network Revenue Managementcited by 3
  18. 2025A Temporal Difference Method for Stochastic Continuous Dynamicscited by 3
  19. 2025Geometry of Neural Reinforcement Learning in Continuous State and Action Spacescited by 3
  20. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 3
  21. 2026ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedulecited by 3
  22. 2026Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithmscited by 3
  23. 2026Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problemscited by 3
  24. 2022Choquet regularization for reinforcement learningcited by 2
  25. 2024Reinforcement Learning for Jump-Diffusions, with Financial Applicationscited by 2
  26. 2024On the grid-sampling limit SDEcited by 2
  27. 2024Polynomial Scaling is Possible For Neural Operator Approximations of Structured Families of BSDEscited by 2
  28. 2024Robust Reinforcement Learning under Diffusion Models for Data with Jumpscited by 2
  29. 2025Continuous-Time Value Iteration for Multi-Agent Reinforcement Learningcited by 2
  30. 2023Discrete-Time Mean-Variance Strategy Based on Reinforcement Learningcited by 1
  31. 2024A Differential and Pointwise Control Approach to Reinforcement Learningcited by 1
  32. 2024A random measure approach to reinforcement learning in continuous timecited by 1
  33. 2025Exploratory Utility Maximization Problem with Tsallis Entropycited by 1
  34. 2026Diffusion Models for Adaptive Sequential Data Generationcited by 1
  35. 2026Actor-Critic Learning for Extended Mean Field Control with Deterministic Policiescited by 1
  36. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 1
  37. 2024On the stability of Lipschitz continuous control problems and its application to reinforcement learningcited by 0
  38. 2025Time After Time: Deep-Q Effect Estimation for Interventions on When and What to docited by 0

50 most cited of 68 All 68 (JSON)