mlsanityData to 14 Oct 2026

Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces

arXiv 2511.06812 ↗First posted 10 Nov 2025math.OC cs.LG math.PRcited by 3

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

We establish the convergence of the deep actor-critic reinforcement learning algorithm presented in [Angiuli et al., 2023a] in the setting of continuous state and action spaces with an infinite discrete-time horizon. This algorithm provides solutions to Mean Field Game (MFG) or Mean Field Control (MFC) problems depending on the ratio between two learning rates: one for the value function and the other for the mean field term. In the MFC case, to rigorously identify the limit, we introduce a discretization of the state and action spaces, following the approach used in the finite-space case in [Angiuli et al., 2023b]. The convergence proofs rely on a generalization of the two-timescale framework introduced in [Borkar, 1997]. We further extend our convergence results to Mean Field Control Games, which involve locally cooperative and globally competitive populations. Finally, we present numerical experiments for linear-quadratic problems in one and two dimensions, for which explicit solutions are available.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 50 references that are in the map. First posted on arXiv 10 Nov 2025.

Builds on

28 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics25 Oct
  2. 2024Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games27 May

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  3. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  4. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  5. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
  6. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 73
  7. 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 62
  8. 2022Learning in Mean Field Games: A Surveycited by 61
  9. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 59
  10. 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 49
  11. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 49
  12. 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 48
14 more
  1. 2020Q-Learning in Regularized Mean-field Gamescited by 42
  2. 2021Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learningcited by 38
  3. 2022Scalable Deep Reinforcement Learning Algorithms for Mean Field Gamescited by 38
  4. 2019Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methodscited by 28
  5. 2019Dynamic Programming Principles for Mean-Field Controls with Learningcited by 23
  6. 2023Actor-Critic learning for mean-field control in continuous timecited by 22
  7. 2018Decentralised Learning in Systems with Many, Many Strategic Agentscited by 20
  8. 2023Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problemscited by 10
  9. 2022Reinforcement Learning Algorithm for Mixed Mean Field Control Gamescited by 7
  10. 2023Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spacescited by 6
  11. 2018Price of Anarchy for Mean Field Gamescited by 3
  12. 2022Reinforcement Learning for Intra-and-Inter-Bank Borrowing and Lending Mean Field Control Gamecited by 3
  13. 2018On the (in)efficiency of MFG equilibriacited by 2
  14. 2023From Nash Equilibrium to Social Optimum and vice versa: a Mean Field Perspectivecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Model-free policy gradient for discrete-time mean-field controlcited by 4
  2. 2025High-dimensional Mean-Field Games by Particle-based Flow Matchingcited by 3
  3. 2026Towards Model-Free Learning in Dynamic Population Games: An Application to Karma Economiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Mean-Field Games through Mean-Field Actor-Critic Flow
  2. 2026Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies
  3. 2024Efficient and Scalable Deep Reinforcement Learning for Mean Field Control Games
  4. 2024Reinforcement Learning for Finite Space Mean-Field Type Games
  5. 2024When is Mean-Field Reinforcement Learning Tractable and Relevant?
  6. 2025Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games
  7. 2026Mean Field Reinforcement Learning
  8. 2026Mean-Field Control on Sparse Graphs: From Local Limits to GNNs via Neighborhood Distributions