# Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games

arXiv: https://arxiv.org/abs/1910.07498 · HTML: https://mlsanity.com/paper/1910.07498/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 19 of 19)

- [Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2107.04050/) (2021)
- [Generalization in Mean Field Games by Learning Master Policies](https://mlsanity.com/paper/2109.09717/) (2021)
- [Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Games](https://mlsanity.com/paper/2212.14449/) (2022)
- [Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Path](https://mlsanity.com/paper/2208.11639/) (2022)
- [Reinforcement Learning for Mean Field Games, with Applications to Economics](https://mlsanity.com/paper/2106.13755/) (2021)
- [Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior](https://mlsanity.com/paper/2307.06175/) (2023)
- [Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies](https://mlsanity.com/paper/2210.04810/) (2022)
- [Regularization of the policy updates for stabilizing Mean Field Games](https://mlsanity.com/paper/2304.01547/) (2023)
- [MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games](https://mlsanity.com/paper/2405.00282/) (2024)
- [Model-free policy gradient for discrete-time mean-field control](https://mlsanity.com/paper/2601.11217/) (2026)
- [Population-size-Aware Policy Optimization for Mean-Field Games](https://mlsanity.com/paper/2302.03364/) (2023)
- [Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics](https://mlsanity.com/paper/2510.22158/) (2025)
- [Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games](https://mlsanity.com/paper/2602.12517/) (2026)
- [Classifying Ambiguous Identities in Hidden-Role Stochastic Games with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.12896/) (2022)
- [Networked Communication for Decentralised Agents in Mean-Field Games](https://mlsanity.com/paper/2306.02766/) (2023)
- [Why does the two-timescale Q-learning converge to different mean field solutions? A unified convergence analysis](https://mlsanity.com/paper/2404.04357/) (2024)
- [Actor-Critic Learning for Risk-Constrained Linear Quadratic Regulation](https://mlsanity.com/paper/2510.22267/) (2025)
- [Data-Driven Network LQG Mean Field Games with Heterogeneous Populations via Integral Reinforcement Learning](https://mlsanity.com/paper/2602.14339/) (2026)
- [Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning](https://mlsanity.com/paper/2610.15871/) (2026)

## Abstract (arXiv)

We study discrete-time mean-field Markov games with infinite numbers of agents where each agent aims to minimize its ergodic cost. We consider the setting where the agents have identical linear state transitions and quadratic cost functions, while the aggregated effect of the agents is captured by the population mean of their states, namely, the mean-field state. For such a game, based on the Nash certainty equivalence principle, we provide sufficient conditions for the existence and uniqueness of its Nash equilibrium. Moreover, to find the Nash equilibrium, we propose a mean-field actor-critic algorithm with linear function approximation, which does not require knowing the model of dynamics. Specifically, at each iteration of our algorithm, we use the single-agent actor-critic algorithm to approximately obtain the optimal policy of the each agent given the current mean-field state, and then update the mean-field state. In particular, we prove that our algorithm converges to the Nash equilibrium at a linear rate. To the best of our knowledge, this is the first success of applying model-free reinforcement learning with function approximation to discrete-time mean-field Markov games with provable non-asymptotic global convergence guarantees.