mlsanityData to 7 Oct 2026

Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games

arXiv 2405.17017 ↗First posted 27 May 2024math.OC cs.LG cs.MAcited by 3

Not on the map. Why?

Abstract (arXiv)

Mean Field Control Games (MFCG), introduced in [Angiuli et al., 2022a], represent competitive games between a large number of large collaborative groups of agents in the infinite limit of number and size of groups. In this paper, we prove the convergence of a three-timescale Reinforcement Q-Learning (RL) algorithm to solve MFCG in a model-free approach from the point of view of representative agents. Our analysis uses a Q-table for finite state and action spaces updated at each discrete time-step over an infinite horizon. In [Angiuli et al., 2023], we proved convergence of two-timescale algorithms for MFG and MFC separately highlighting the need to follow multiple population distributions in the MFC case. Here, we integrate this feature for MFCG as well as three rates of update decreasing to zero in the proper ratios. Our technique of proof uses a generalization to three timescales of the two-timescale analysis in [Borkar, 1997]. We give a simple example satisfying the various hypothesis made in the proof of convergence and illustrating the performance of the algorithm.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 13 of its 38 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

13 of 38 references.

References · 13

  1. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 65
  2. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 56
  3. 2022Learning in Mean Field Games: A Surveycited by 56
  4. 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 45
  5. 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 44
  6. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 42
  7. 2019Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methodscited by 24
  8. 2019Mean-field Markov decision processes with common noise and open-loop controlscited by 24
  9. 2023Actor-Critic learning for mean-field control in continuous timecited by 19
  10. 2023Continuous-time q-learning for mean-field control problemscited by 17
  11. 2022Reinforcement Learning Algorithm for Mixed Mean Field Control Gamescited by 7
  12. 2023Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problemscited by 7
1 more
  1. 2022Reinforcement Learning for Intra-and-Inter-Bank Borrowing and Lending Mean Field Control Gamecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spacescited by 3
  2. 2024Last Iterate Convergence in Monotone Mean Field Gamescited by 0
  3. 2026Robust Q-learning for mean-field control under Wasserstein uncertainty in common noisecited by 0