# Learning Equilibria in Mean-Field Games: Introducing Mean-Field PSRO

arXiv: https://arxiv.org/abs/2111.08350 · HTML: https://mlsanity.com/paper/2111.08350/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2021.

## Built on by (most cited first, 14 of 14)

- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)
- [A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.03859/) (2022)
- [Policy Space Response Oracles: A Survey](https://mlsanity.com/paper/2403.02227/) (2024)
- [Learning Correlated Equilibria in Mean-Field Games](https://mlsanity.com/paper/2208.10138/) (2022)
- [MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games](https://mlsanity.com/paper/2405.00282/) (2024)
- [Major-Minor Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.10665/) (2023)
- [Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environments](https://mlsanity.com/paper/2209.10958/) (2022)
- [Self-adaptive PSRO: Towards an Automatic Population-based Game Solver](https://mlsanity.com/paper/2404.11144/) (2024)
- [Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games](https://mlsanity.com/paper/2602.12517/) (2026)
- [Mean Field Correlated Imitation Learning](https://mlsanity.com/paper/2404.09324/) (2024)
- [Bounded Rationality Equilibrium Learning in Mean Field Games](https://mlsanity.com/paper/2411.07099/) (2024)
- [The Hidden Game Problem](https://mlsanity.com/paper/2510.03845/) (2025)
- [NePPO: Near-Potential Policy Optimization for General-Sum Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2603.06977/) (2026)
- [Optimal Coarse Correlated Equilibria in Mean Field Games: Linear Programming and No-Regret Learning](https://mlsanity.com/paper/2606.20062/) (2026)

## Abstract (arXiv)

Recent advances in multiagent learning have seen the introduction ofa family of algorithms that revolve around the population-based trainingmethod PSRO, showing convergence to Nash, correlated and coarse corre-lated equilibria. Notably, when the number of agents increases, learningbest-responses becomes exponentially more difficult, and as such ham-pers PSRO training methods. The paradigm of mean-field games pro-vides an asymptotic solution to this problem when the considered gamesare anonymous-symmetric. Unfortunately, the mean-field approximationintroduces non-linearities which prevent a straightforward adaptation ofPSRO. Building upon optimization and adversarial regret minimization,this paper sidesteps this issue and introduces mean-field PSRO, an adap-tation of PSRO which learns Nash, coarse correlated and correlated equi-libria in mean-field games. The key is to replace the exact distributioncomputation step by newly-defined mean-field no-adversarial-regret learn-ers, or by black-box optimization. We compare the asymptotic complexityof the approach to standard PSRO, greatly improve empirical bandit con-vergence speed by compressing temporal mixture weights, and ensure itis theoretically robust to payoff noise. Finally, we illustrate the speed andaccuracy of mean-field PSRO on several mean-field games, demonstratingconvergence to strong and weak equilibria.