mlsanityData to 14 Oct 2026

Last Iterate Convergence in Monotone Mean Field Games

arXiv 2410.05127 ↗First posted 7 Oct 2024cs.GT cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In the Lasry--Lions framework, Mean-Field Games (MFGs) model interactions among an infinite number of agents. However, existing algorithms either require strict monotonicity or only guarantee the convergence of averaged iterates, as in Fictitious Play in continuous time. We address this gap with the following theoretical result. First, we prove that the last-iterated policy of a proximal-point (PP) update with KL regularization converges to an equilibrium of MFG under non-strict monotonicity. Second, we see that each PP update is equivalent to finding the equilibria of a KL-regularized MFG. We then prove that this equilibrium can be found using Mirror Descent (MD) with an exponential last-iterate convergence rate. Building on these insights, we propose the Approximate Proximal-Point (mathttAPP) algorithm, which approximately implements the PP update via a small number of MD steps. Numerical experiments on standard benchmarks confirm that the mathttAPP algorithm reliably converges to the unregularized mean-field equilibrium without time-averaging.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 70 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

42 of 70 references.

References · 42

  1. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  2. 2019A Theory of Regularized Markov Decision Processescited by 286
  3. 2018Mean Field Multi-Agent Reinforcement Learningcited by 207
  4. 2017Cycles in adversarial regularized learningcited by 124
  5. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
  6. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 73
  7. 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 62
  8. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 61
  9. 2022Learning in Mean Field Games: A Surveycited by 61
  10. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  11. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 49
  12. 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 48
30 more
  1. 2020Q-Learning in Regularized Mean-field Gamescited by 42
  2. 2015Learning in Mean Field Games: the Fictitious Playcited by 38
  3. 2022Scalable Deep Reinforcement Learning Algorithms for Mean Field Gamescited by 38
  4. 2016Markov-Nash Equilibria in Mean-Field Games with Discounted Costcited by 31
  5. 2021Generalization in Mean Field Games by Learning Master Policiescited by 28
  6. 2021Learning Graphon Mean Field Games and Approximate Nash Equilibriacited by 28
  7. 2022Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Gamescited by 26
  8. 2022Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Gamescited by 26
  9. 2021Numerical Methods for Mean Field Games and Mean Field Type Controlcited by 22
  10. 2021Exploration-Exploitation in Multi-Agent Competition: Convergence with Bounded Rationalitycited by 21
  11. 2021Beyond Time-Average Convergence: Near-Optimal Uncoupled Online Learning via Clairvoyant Multiplicative Weights Updatecited by 20
  12. 2022Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Pathcited by 20
  13. 2022A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learningcited by 14
  14. 2022Learning Sparse Graphon Mean Field Gamescited by 14
  15. 2023On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximationcited by 14
  16. 2023Wasserstein Mirror Gradient Flow as the limit of the Sinkhorn Algorithmcited by 10
  17. 2018Finite mean field games: fictitious play and convergence to a first order continuous mean field gamecited by 9
  18. 2022MF-OMO: An Optimization Formulation of Mean-Field Gamescited by 9
  19. 2022Last-Iterate Convergence with Full and Noisy Feedback in Two-Player Zero-Sum Gamescited by 9
  20. 2023Adaptively Perturbed Mirror Descent for Learning in Gamescited by 9
  21. 2023Learning Regularized Monotone Graphon Mean-Field Gamescited by 9
  22. 2024When is Mean-Field Reinforcement Learning Tractable and Relevant?cited by 8
  23. 2019Value Iteration Algorithm for Mean-field Gamescited by 5
  24. 2022Asynchronous Gradient Play in Zero-Sum Multi-agent Gamescited by 5
  25. 2023Variational Principles for Mirror Descent and Mirror Langevin Dynamicscited by 5
  26. 2024MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Gamescited by 5
  27. 2024Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximationcited by 5
  28. 2024Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Gamescited by 3
  29. 2024Exploiting Approximate Symmetry for Efficient Multi-Agent Reinforcement Learningcited by 2
  30. 2022A fictitious-play finite-difference method for linearly solvable mean field gamescited by 1

Built on by

Not cited yet.