mlsanityData to 7 Oct 2026

Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations

arXiv 2604.27372 ↗First posted 30 Apr 2026math.OC cs.LG cs.MAcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper investigates the continuous-time counterpart of the Q-function for entropy-regularized mean-field control (MFC) with controlled common noise, coined as q-function by Jia and Zhou (2023) in the single agent's model. We first show that, under discretely sampled actions, the value function in the exploratory formulation converges to the one in the relaxed control formulation as the time grid refines. Leveraging the relaxed control formulation, we derive the exploratory Hamilton-Jacobi-Bellman (HJB) equation, in which the controlled common noise gives rise to an additional nonlinear functional of policy, rendering the policy iteration intricate. Under certain concavity condition, we establish the existence and uniqueness of the optimal one-step policy iteration via a first-order condition using the partial linear functional derivative with respect to policy. The policy improvement at each iteration is verified by relating to an entropy-regularized optimization problem over the space of policies. In the mean-field setting, we introduce the integrated q-function (Iq-function) defined on the state distribution and the policy, and it is shown that an optimal policy is identified as a two-layer fixed point to the argmax operator of the Iq-function. Finally, we provide the explicit characterization of an optimal policy as a Gaussian distribution in the general linear-quadratic (LQ) setting.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 26 of its 51 references that are in the map. First posted on arXiv 30 Apr 2026.

Builds on

26 of 51 references.

References · 26

  1. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  2. 2022q-Learning in Continuous Timecited by 68
  3. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
  4. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 56
  5. 2020Entropy Regularization for Mean Field Games with Learningcited by 50
  6. 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 44
  7. 2019Mean-field Markov decision processes with common noise and open-loop controlscited by 24
  8. 2019Dynamic Programming Principles for Mean-Field Controls with Learningcited by 22
  9. 2023Actor-Critic learning for mean-field control in continuous timecited by 19
  10. 2023Continuous-time q-learning for mean-field control problemscited by 17
  11. 2022Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithmcited by 12
  12. 2016Dynamic programming for optimal control of stochastic McKean-Vlasov dynamicscited by 11
14 more
  1. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learningcited by 9
  2. 2020Game on Random Environment, Mean-field Langevin System and Neural Networkscited by 8
  3. 2023On optimal tracking portfolio in incomplete markets: The reinforcement learning approachcited by 7
  4. 2024Unified continuous-time q-learning for mean-field game and mean-field control problemscited by 7
  5. 2016Linear Quadratic Mean Field Type Control and Mean Field Games with Common Noise, with Application to Production of an Exhaustible Resourcecited by 6
  6. 2019McKean-Vlasov optimal control: the dynamic programming principlecited by 6
  7. 2023Data-Driven Merton's Strategies via Policy Randomizationcited by 6
  8. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 5
  9. 2014A Probabilistic approach to classical solutions of the master equation for large population equilibriacited by 4
  10. 2025Continuous-time reinforcement learning for optimal switching over multiple regimescited by 4
  11. 2024On the grid-sampling limit SDEcited by 2
  12. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Controlcited by 2
  13. 2023A Viscosity Solution Theory of Stochastic Hamilton-Jacobi-Bellman equations in the Wasserstein Spacecited by 1
  14. 2024Viscosity Solutions for HJB Equations on the Process Spacecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Actor-Critic Learning for Extended Mean Field Control with Deterministic Policiescited by 1
  2. 2026Robust Q-learning for mean-field control under Wasserstein uncertainty in common noisecited by 0
  3. 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Datacited by 0
  4. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  5. 2026Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradientcited by 0