mlsanityData to 7 Oct 2026

Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms

arXiv 2604.27378 ↗First posted 30 Apr 2026math.OC cs.LG cs.MAcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper is a continuation work of Ren et al. (2026) aiming to further devise q-learning algorithms for mean-field control (MFC) with controlled common noise. Based on the relaxed control formulation, we first establish the martingale condition of the value function and the Iq-function by evaluating along the conditional state distributions generated by all test policies. As the data in the relaxed control formulation are not observable in practice, we quantify the error incurred when they are replaced by the observable ones in the exploratory formulation under discretely sampled actions. This, together with a two-layer fixed point characterization of an optimal policy in Ren et al. (2026), allows us to propose several algorithms including the Actor-Critic q-learning algorithm, in which the policy is updated in the Actor-step based on the iteration rule induced by the improved Iq-function, and the value function and Iq-function are updated in the Critic-step based on the martingale orthogonality condition using the data from the exploratory formulation. We also establish the convergence of the inner iterations in the Actor-step in an infinite-horizon linear quadratic (LQ) framework. In two examples, within and beyond LQ framework, our q-learning algorithms are implemented with satisfactory performance.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 30 Apr 2026.

Builds on

22 of 29 references.

References · 22

  1. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  2. 2022q-Learning in Continuous Timecited by 68
  3. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
  4. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 56
  5. 2022Learning in Mean Field Games: A Surveycited by 56
  6. 2020Entropy Regularization for Mean Field Games with Learningcited by 50
  7. 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 44
  8. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 42
  9. 2021Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approachcited by 27
  10. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 19
  11. 2023Actor-Critic learning for mean-field control in continuous timecited by 19
  12. 2023Continuous-time q-learning for mean-field control problemscited by 17
10 more
  1. 2022Mean-field neural networks: learning mappings on Wasserstein spacecited by 15
  2. 2022Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithmcited by 12
  3. 2022Mean-field neural networks-based algorithms for McKean-Vlasov control problems *cited by 9
  4. 2021Discrete-Time Mean Field Control with Environment Statescited by 7
  5. 2023On optimal tracking portfolio in incomplete markets: The reinforcement learning approachcited by 7
  6. 2024Unified continuous-time q-learning for mean-field game and mean-field control problemscited by 7
  7. 2023Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spacescited by 6
  8. 2023Data-Driven Merton's Strategies via Policy Randomizationcited by 6
  9. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 5
  10. 2025Continuous-time reinforcement learning for optimal switching over multiple regimescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Actor-Critic Learning for Extended Mean Field Control with Deterministic Policiescited by 1
  2. 2026Robust Q-learning for mean-field control under Wasserstein uncertainty in common noisecited by 0
  3. 2026Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradientcited by 0