mlsanityData to 14 Oct 2026

Why does the two-timescale Q-learning converge to different mean field solutions? A unified convergence analysis

arXiv 2404.04357 ↗First posted 5 Apr 2024math.OCcited by 0

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

We revisit the unified two-timescale Q-learning algorithm as initially introduced by Angiuli et al. citeangiuli2022unified. This algorithm demonstrates efficacy in solving mean field game (MFG) and mean field control (MFC) problems, simply by tuning the ratio of two learning rates for mean field distribution and the Q-functions respectively. In this paper, we provide a comprehensive theoretical explanation of the algorithm's bifurcated numerical outcomes under fixed learning rates. We achieve this by establishing a diagram that correlates continuous-time mean field problems to their discrete-time Q-function counterparts, forming the basis of the algorithm. Our key contribution lies in the construction of a Lyapunov function integrating both mean field distribution and Q-function iterates. This Lyapunov function facilitates a unified convergence of the algorithm across the entire spectrum of learning rates, thus providing a cohesive framework for analysis.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 52 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

28 of 52 references.

References · 28

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  3. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  4. 2018Mean Field Multi-Agent Reinforcement Learningcited by 207
  5. 2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 177
  6. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
  7. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  8. 2022q-Learning in Continuous Timecited by 91
  9. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 73
  10. 2022Learning in Mean Field Games: A Surveycited by 61
  11. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 59
  12. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
16 more
  1. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  2. 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 49
  3. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 49
  4. 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 31
  5. 2019Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methodscited by 28
  6. 2021A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learningcited by 24
  7. 2023Recent Developments in Machine Learning Methods for Stochastic Control and Gamescited by 24
  8. 2018Decentralised Learning in Systems with Many, Many Strategic Agentscited by 20
  9. 2020Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Timecited by 20
  10. 2022Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Pathcited by 20
  11. 2019Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Gamescited by 19
  12. 2021Reinforcement Learning for Mean Field Games, with Applications to Economicscited by 15
  13. 2022Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guaranteescited by 14
  14. 2023Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problemscited by 10
  15. 2019Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Timecited by 9
  16. 2013Global Adaptive Dynamic Programming for Continuous-Time Nonlinear Systemscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified continuous-time q-learning for mean-field game and mean-field control problems
  2. 2024Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games
  3. 2026Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms
  4. 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations
  5. 2025Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces
  6. 2026Robust Q-learning for mean-field control under Wasserstein uncertainty in common noise
  7. 2026Mean Field Reinforcement Learning
  8. 2024Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise