# Scaling up Mean Field Games with Online Mirror Descent

arXiv: https://arxiv.org/abs/2103.00623 · HTML: https://mlsanity.com/paper/2103.00623/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 82 references that are in the map.
- First posted on arXiv 28 Feb 2021.

## Builds on (most cited first, 27 of 27)

- [DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker](https://mlsanity.com/paper/1701.01724/) (2017)
- [Cycles in adversarial regularized learning](https://mlsanity.com/paper/1709.02738/) (2017)
- [A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems](https://mlsanity.com/paper/1912.01825/) (2019)
- [Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications](https://mlsanity.com/paper/2007.03458/) (2020)
- [Unified Reinforcement Q-Learning for Mean Field Game and Control Problems](https://mlsanity.com/paper/2006.13912/) (2020)
- [On the Convergence of Model Free Learning in Mean Field Games](https://mlsanity.com/paper/1907.02633/) (2019)
- [Q-Learning in Regularized Mean-field Games](https://mlsanity.com/paper/2003.12151/) (2020)
- [Learning in Mean Field Games: the Fictitious Play](https://mlsanity.com/paper/1507.06280/) (2015)
- [Adapting a Kidney Exchange Algorithm to Align with Human Values](https://mlsanity.com/paper/2005.09755/) (2020)
- [Bandit learning in concave N-person games](https://mlsanity.com/paper/1810.01925/) (2018)
- [Convergence Analysis of Machine Learning Algorithms for the Numerical Solution of Mean Field Control and Games: II -- The Finite Horizon Case](https://mlsanity.com/paper/1908.01613/) (2019)
- [Multiplicative Weights Update with Constant Step-Size in Congestion Games: Convergence, Limit Cycles and Chaos](https://mlsanity.com/paper/1703.01138/) (2017)
- [Deep Learning Methods for Mean Field Control Problems with Delay](https://mlsanity.com/paper/1905.00358/) (2019)
- [Mean Field Games and Applications: Numerical Aspects](https://mlsanity.com/paper/2003.04444/) (2020)
- [Numerical resolution of McKean-Vlasov FBSDEs using neural networks](https://mlsanity.com/paper/1909.12678/) (2019)
- [The route to chaos in routing games: When is Price of Anarchy too optimistic?](https://mlsanity.com/paper/1906.02486/) (2019)
- [Proximal methods for stationary Mean Field Games with local couplings](https://mlsanity.com/paper/1608.07701/) (2016)
- [Learning in anonymous nonatomic games with applications to first-order mean field games](https://mlsanity.com/paper/1704.00378/) (2017)
- [On the implementation of a primal-dual algorithm for second order time-dependent mean field games with local couplings](https://mlsanity.com/paper/1802.07902/) (2018)
- [Finite mean field games: fictitious play and convergence to a first order continuous mean field game](https://mlsanity.com/paper/1805.05940/) (2018)
- [Connecting GANs, MFGs, and OT](https://mlsanity.com/paper/2002.04112/) (2020)
- [Model-free Reinforcement Learning for Non-stationary Mean Field Games](https://mlsanity.com/paper/2004.02073/) (2020)
- [Provable Fictitious Play for General Mean-Field Games](https://mlsanity.com/paper/2010.04211/) (2020)
- [Mean-field type modeling of nonlocal crowd aversion in pedestrian crowd dynamics](https://mlsanity.com/paper/1701.09118/) (2017)
- [Mean Field Control and Mean Field Game Models with Several Populations](https://mlsanity.com/paper/1810.00783/) (2018)
- [On the system of partial differential equations arising in mean field type control](https://mlsanity.com/paper/1503.05044/) (2015)
- [Convergence of some Mean Field Games systems to aggregation and flocking models](https://mlsanity.com/paper/2004.04403/) (2020)

## Built on by (most cited first, 30 of 32)

- [Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint](https://mlsanity.com/paper/2106.03787/) (2021)
- [Scalable Deep Reinforcement Learning Algorithms for Mean Field Games](https://mlsanity.com/paper/2203.11973/) (2022)
- [Generalization in Mean Field Games by Learning Master Policies](https://mlsanity.com/paper/2109.09717/) (2021)
- [Learning Graphon Mean Field Games and Approximate Nash Equilibria](https://mlsanity.com/paper/2112.01280/) (2021)
- [Reinforcement Learning for Mean Field Games, with Applications to Economics](https://mlsanity.com/paper/2106.13755/) (2021)
- [Learning Sparse Graphon Mean Field Games](https://mlsanity.com/paper/2209.03880/) (2022)
- [On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation](https://mlsanity.com/paper/2305.11283/) (2023)
- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Learning Regularized Monotone Graphon Mean-Field Games](https://mlsanity.com/paper/2310.08089/) (2023)
- [Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL](https://mlsanity.com/paper/2402.05724/) (2024)
- [Learning Correlated Equilibria in Mean-Field Games](https://mlsanity.com/paper/2208.10138/) (2022)
- [Regularization of the policy updates for stabilizing Mean Field Games](https://mlsanity.com/paper/2304.01547/) (2023)
- [MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games](https://mlsanity.com/paper/2405.00282/) (2024)
- [Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation](https://mlsanity.com/paper/2408.08192/) (2024)
- [Learning Regularized Graphon Mean-Field Games with Unknown Graphons](https://mlsanity.com/paper/2310.17531/) (2023)
- [On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning](https://mlsanity.com/paper/2411.15046/) (2024)
- [Hypergraphon Mean Field Games](https://mlsanity.com/paper/2203.16223/) (2022)
- [Population-size-Aware Policy Optimization for Mean-Field Games](https://mlsanity.com/paper/2302.03364/) (2023)
- [Learning in Herding Mean Field Games: Single-Loop Algorithm with Finite-Time Convergence Analysis](https://mlsanity.com/paper/2408.04780/) (2024)
- [Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environments](https://mlsanity.com/paper/2209.10958/) (2022)
- [A Single Online Agent Can Efficiently Learn Mean Field Games](https://mlsanity.com/paper/2405.03718/) (2024)
- [MESOB: Balancing Equilibria & Social Optimality](https://mlsanity.com/paper/2307.07911/) (2023)
- [Reinforcement Learning for SBM Graphon Games with Re-Sampling](https://mlsanity.com/paper/2310.16326/) (2023)
- [Steering No-Regret Agents in MFGs under Model Uncertainty](https://mlsanity.com/paper/2503.09309/) (2025)
- [Reimagining Demand-Side Management with Mean Field Learning](https://mlsanity.com/paper/2302.08190/) (2023)
- [Networked Communication for Decentralised Agents in Mean-Field Games](https://mlsanity.com/paper/2306.02766/) (2023)
- [Networked Communication for Mean-Field Games with Function Approximation and Empirical Mean-Field Estimation](https://mlsanity.com/paper/2408.11607/) (2024)
- [Vulnerable Agent Identification in Large-Scale Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2509.15103/) (2025)
- [Recurrent Structural Policy Gradient for Partially Observable Mean Field Games](https://mlsanity.com/paper/2602.20141/) (2026)
- [Towards Model-Free Learning in Dynamic Population Games: An Application to Karma Economies](https://mlsanity.com/paper/2605.11042/) (2026)

## Abstract (arXiv)

We address scaling up equilibrium computation in Mean Field Games (MFGs) using Online Mirror Descent (OMD). We show that continuous-time OMD provably converges to a Nash equilibrium under a natural and well-motivated set of monotonicity assumptions. This theoretical result nicely extends to multi-population games and to settings involving common noise. A thorough experimental investigation on various single and multi-population MFGs shows that OMD outperforms traditional algorithms such as Fictitious Play (FP). We empirically show that OMD scales up and converges significantly faster than FP by solving, for the first time to our knowledge, examples of MFGs with hundreds of billions states. This study establishes the state-of-the-art for learning in large-scale multi-agent and multi-population games.