mlsanityData to 14 Oct 2026

Continuous-time q-learning for mean-field control problems

arXiv 2306.16208 ↗First posted 28 Jun 2023cs.LG math.OC q-fin.CPcited by 22

Not on the map. Why?

Cited by 20 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper studies the q-learning, recently coined as the continuous time counterpart of Q-learning by Jia and Zhou (2023), for continuous time Mckean-Vlasov control problems in the setting of entropy-regularized reinforcement learning. In contrast to the single agent's control problem in Jia and Zhou (2023), the mean-field interaction of agents renders the definition of the q-function more subtle, for which we reveal that two distinct q-functions naturally arise: (i) the integrated q-function (denoted by q) as the first-order approximation of the integrated Q-function introduced in Gu, Guo, Wei and Xu (2023), which can be learnt by a weak martingale condition involving test policies; and (ii) the essential q-function (denoted by qe) that is employed in the policy improvement iterations. We show that two q-functions are related via an integral representation under all test policies. Based on the weak martingale condition and our proposed searching method of test policies, some model-free learning algorithms are devised. In two examples, one in LQ control framework and one beyond LQ control framework, we can obtain the exact parameterization of the optimal value function and q-functions and illustrate our algorithms with simulation experiments.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

20 of 35 references.

References · 20

  1. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  2. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  3. 2022q-Learning in Continuous Timecited by 91
  4. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  5. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 59
  6. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  7. 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 49
  8. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 49
  9. 2021Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learningcited by 38
  10. 2021On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)cited by 32
  11. 2021Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approachcited by 28
  12. 2019Mean-field Markov decision processes with common noise and open-loop controlscited by 25
8 more
  1. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 25
  2. 2021Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controlscited by 24
  3. 2019Dynamic Programming Principles for Mean-Field Controls with Learningcited by 23
  4. 2023Actor-Critic learning for mean-field control in continuous timecited by 22
  5. 2021Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex modelscited by 21
  6. 2016Limit theory for controlled McKean-Vlasov dynamicscited by 12
  7. 2023Mean-Field Control based Approximation of Multi-Agent Reinforcement Learning in Presence of a Non-decomposable Shared Global Statecited by 8
  8. 2020Optimal control of path-dependent McKean-Vlasov SDEs in infinite dimensioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 12.

22 papers.

Sort:

By area:

  1. 2024Unified continuous-time q-learning for mean-field game and mean-field control problemscited by 13
  2. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learningcited by 13
  3. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problemscited by 12
  4. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 9
  5. 2024Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropycited by 8
  6. 2025Continuous-time reinforcement learning for optimal switching over multiple regimescited by 6
  7. 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundationscited by 6
  8. 2026Model-free policy gradient for discrete-time mean-field controlcited by 4
  9. 2026Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithmscited by 4
  10. 2026Policy Gradient for Continuous-Time Mean-Field Controlcited by 4
  11. 2024Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Gamescited by 3
  12. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Controlcited by 3
10 more
  1. 2026Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problemscited by 3
  2. 2025A Reinforcement Learning Framework for Some Singular Stochastic Control Problemscited by 2
  3. 2025Reinforcement learning for irreversible reinsurance problems: the randomized singular control approachcited by 1
  4. 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Datacited by 1
  5. 2026Actor-Critic Learning for Extended Mean Field Control with Deterministic Policiescited by 1
  6. 2026An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flowscited by 0
  7. 2026Robust Q-learning for mean-field control under Wasserstein uncertainty in common noisecited by 0
  8. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  9. 2026Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policiescited by 0
  10. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergencecited by 0