# Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations arXiv: https://arxiv.org/abs/2604.27372 ยท HTML: https://mlsanity.com/paper/2604.27372/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. - Builds on 26 of its 51 references that are in the map. - First posted on arXiv 30 Apr 2026. ## Placed in - Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/) ## Builds on (most cited first, 26 of 26) - [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021) - [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022) - [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021) - [Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning](https://mlsanity.com/paper/1910.12802/) (2019) - [Entropy Regularization for Mean Field Games with Learning](https://mlsanity.com/paper/2010.00145/) (2020) - [Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis](https://mlsanity.com/paper/2002.04131/) (2020) - [Mean-field Markov decision processes with common noise and open-loop controls](https://mlsanity.com/paper/1912.07883/) (2019) - [Dynamic Programming Principles for Mean-Field Controls with Learning](https://mlsanity.com/paper/1911.07314/) (2019) - [Actor-Critic learning for mean-field control in continuous time](https://mlsanity.com/paper/2303.06993/) (2023) - [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023) - [Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.02409/) (2022) - [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024) - [Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2503.09981/) (2025) - [Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics](https://mlsanity.com/paper/1604.04057/) (2016) - [On optimal tracking portfolio in incomplete markets: The reinforcement learning approach](https://mlsanity.com/paper/2311.14318/) (2023) - [Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty](https://mlsanity.com/paper/2404.12598/) (2024) - [Game on Random Environment, Mean-field Langevin System and Neural Networks](https://mlsanity.com/paper/2004.02457/) (2020) - [Data-Driven Merton's Strategies via Policy Randomization](https://mlsanity.com/paper/2312.11797/) (2023) - [Linear Quadratic Mean Field Type Control and Mean Field Games with Common Noise, with Application to Production of an Exhaustible Resource](https://mlsanity.com/paper/1607.02130/) (2016) - [McKean-Vlasov optimal control: the dynamic programming principle](https://mlsanity.com/paper/1907.08860/) (2019) - [Continuous-time reinforcement learning for optimal switching over multiple regimes](https://mlsanity.com/paper/2512.04697/) (2025) - [A Probabilistic approach to classical solutions of the master equation for large population equilibria](https://mlsanity.com/paper/1411.3009/) (2014) - [On the grid-sampling limit SDE](https://mlsanity.com/paper/2410.07778/) (2024) - [Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control](https://mlsanity.com/paper/2504.21793/) (2025) - [A Viscosity Solution Theory of Stochastic Hamilton-Jacobi-Bellman equations in the Wasserstein Space](https://mlsanity.com/paper/2310.14446/) (2023) - [Viscosity Solutions for HJB Equations on the Process Space](https://mlsanity.com/paper/2401.04920/) (2024) ## Built on by (most cited first, 6 of 6) - [Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data](https://mlsanity.com/paper/2606.26498/) (2026) - [Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies](https://mlsanity.com/paper/2607.11005/) (2026) - [Robust Q-learning for mean-field control under Wasserstein uncertainty in common noise](https://mlsanity.com/paper/2606.20356/) (2026) - [Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning](https://mlsanity.com/paper/2607.03168/) (2026) - [Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policies](https://mlsanity.com/paper/2607.19928/) (2026) - [Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradient](https://mlsanity.com/paper/2610.09712/) (2026) ## Abstract (arXiv) This paper investigates the continuous-time counterpart of the Q-function for entropy-regularized mean-field control (MFC) with controlled common noise, coined as q-function by Jia and Zhou (2023) in the single agent's model. We first show that, under discretely sampled actions, the value function in the exploratory formulation converges to the one in the relaxed control formulation as the time grid refines. Leveraging the relaxed control formulation, we derive the exploratory Hamilton-Jacobi-Bellman (HJB) equation, in which the controlled common noise gives rise to an additional nonlinear functional of policy, rendering the policy iteration intricate. Under certain concavity condition, we establish the existence and uniqueness of the optimal one-step policy iteration via a first-order condition using the partial linear functional derivative with respect to policy. The policy improvement at each iteration is verified by relating to an entropy-regularized optimization problem over the space of policies. In the mean-field setting, we introduce the integrated q-function (Iq-function) defined on the state distribution and the policy, and it is shown that an optimal policy is identified as a two-layer fixed point to the argmax operator of the Iq-function. Finally, we provide the explicit characterization of an optimal policy as a Gaussian distribution in the general linear-quadratic (LQ) setting.