# Provable Fictitious Play for General Mean-Field Games

arXiv: https://arxiv.org/abs/2010.04211 · HTML: https://mlsanity.com/paper/2010.04211/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2020.

## Built on by (most cited first, 7 of 7)

- [Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint](https://mlsanity.com/paper/2106.03787/) (2021)
- [Softmax Policy Gradient Methods Can Take Exponential Time to Converge](https://mlsanity.com/paper/2102.11270/) (2021)
- [Scalable Deep Reinforcement Learning Algorithms for Mean Field Games](https://mlsanity.com/paper/2203.11973/) (2022)
- [Scaling up Mean Field Games with Online Mirror Descent](https://mlsanity.com/paper/2103.00623/) (2021)
- [Reinforcement Learning for Mean Field Games, with Applications to Economics](https://mlsanity.com/paper/2106.13755/) (2021)
- [Learning High-Dimensional McKean-Vlasov Forward-Backward Stochastic Differential Equations with General Distribution Dependence](https://mlsanity.com/paper/2204.11924/) (2022)
- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)

## Abstract (arXiv)

We propose a reinforcement learning algorithm for stationary mean-field games, where the goal is to learn a pair of mean-field state and stationary policy that constitutes the Nash equilibrium. When viewing the mean-field state and the policy as two players, we propose a fictitious play algorithm which alternatively updates the mean-field state and the policy via gradient-descent and proximal policy optimization, respectively. Our algorithm is in stark contrast with previous literature which solves each single-agent reinforcement learning problem induced by the iterates mean-field states to the optimum. Furthermore, we prove that our fictitious play algorithm converges to the Nash equilibrium at a sublinear rate. To the best of our knowledge, this seems the first provably convergent single-loop reinforcement learning algorithm for mean-field games based on iterative updates of both mean-field state and policy.