mlsanityData to 7 Oct 2026

Unified continuous-time q-learning for mean-field game and mean-field control problems

arXiv 2407.04521 ↗First posted 5 Jul 2024math.OC cs.LG q-fin.CPcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper studies the continuous-time q-learning in mean-field jump-diffusion models in a setting where the environment simulator does not provide direct access to the population distribution. We propose the integrated q-function in decoupled form (decoupled Iq-function) and establish its martingale characterization, which provides a unified policy evaluation rule for both mean-field game (MFG) and mean-field control (MFC) problems. Moreover, we consider the learning procedure where population distribution is updated based on the representative agent's state values. Depending on the task to solve the MFG or MFC problem, we can employ the decoupled Iq-function differently to characterize the mean-field equilibrium policy or the mean-field optimal policy respectively. Based on these theoretical findings, we devise a unified parametric q-learning algorithm for both MFG and MFC problems by utilizing test policies and the averaged martingale orthogonality condition. In two applications within and beyond LQ framework, we illustrate the effectiveness and efficiency of our unified parametric q-learning algorithm for both MFG and MFC learning tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 41 of its 70 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

41 of 70 references.

References · 41

  1. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  2. 2022q-Learning in Continuous Timecited by 68
  3. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 65
  4. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
  5. 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 57
  6. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 56
  7. 2022Learning in Mean Field Games: A Surveycited by 56
  8. 2020Entropy Regularization for Mean Field Games with Learningcited by 50
  9. 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 45
  10. 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 44
  11. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 42
  12. 2020Q-Learning in Regularized Mean-field Gamescited by 40
29 more
  1. 2021Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learningcited by 38
  2. 2021On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)cited by 31
  3. 2016Markov-Nash Equilibria in Mean-Field Games with Discounted Costcited by 29
  4. 2021Scaling up Mean Field Games with Online Mirror Descentcited by 29
  5. 2021Generalization in Mean Field Games by Learning Master Policiescited by 28
  6. 2021Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approachcited by 27
  7. 2019Dynamic Programming Principles for Mean-Field Controls with Learningcited by 22
  8. 2023Actor-Critic learning for mean-field control in continuous timecited by 19
  9. 2023Continuous-time q-learning for mean-field control problemscited by 17
  10. 2019Stochastic Graphon Games: I. The Static Casecited by 12
  11. 2022Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithmcited by 12
  12. 2016Limit theory for controlled McKean-Vlasov dynamicscited by 11
  13. 2020Exploratory LQG Mean Field Games with Entropy Regularizationcited by 11
  14. 2017Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactionscited by 8
  15. 2023Mean-Field Control based Approximation of Multi-Agent Reinforcement Learning in Presence of a Non-decomposable Shared Global Statecited by 8
  16. 2020Provable Fictitious Play for General Mean-Field Gamescited by 7
  17. 2021Discrete-Time Mean Field Control with Environment Statescited by 7
  18. 2023On optimal tracking portfolio in incomplete markets: The reinforcement learning approachcited by 7
  19. 2023Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problemscited by 7
  20. 2024Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropycited by 7
  21. 2023Actor critic learning algorithms for mean-field control with moment neural networkscited by 6
  22. 2023Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spacescited by 6
  23. 2021Mean Field Games Master Equations with Non-separable Hamiltonians and Displacement Monotonicitycited by 4
  24. 2023Reinforcement Learning with Function Approximation: From Linear to Nonlinearcited by 4
  25. 2020McKean-Vlasov optimal control: limit theory and equivalence between different formulationscited by 2
  26. 2021Optimal control of the Fokker-Planck equation under state constraints in the Wasserstein spacecited by 2
  27. 2023Reinforcement Learning for optimal dividend problem under diffusion modelcited by 2
  28. 2023From Nash Equilibrium to Social Optimum and vice versa: a Mean Field Perspectivecited by 2
  29. 2024Optimal Control of the Nonlinear Stochastic Fokker--Planck Equationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundationscited by 5
  2. 2025Continuous-time reinforcement learning for optimal switching over multiple regimescited by 4
  3. 2026Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithmscited by 3
  4. 2026Actor-Critic Learning for Extended Mean Field Control with Deterministic Policiescited by 1
  5. 2026Robust Q-learning for mean-field control under Wasserstein uncertainty in common noisecited by 0
  6. 2026Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricingcited by 0
  7. 2026Boundary-aware Reinforcement Learning for Hypercube State Spaces via Deterministic Policy Gradientcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control
  2. 2025Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensembles
  3. 2022Individual-Level Inverse Reinforcement Learning for Mean Field Games
  4. 2023Boosting Continuous Control with Consistency Policy
  5. 2024Maximum Causal Entropy IRL in Mean-Field Games and GNEP Framework for Forward RL
  6. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization
  7. 2026Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward
  8. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint