# Mean Field Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/1802.05438 · HTML: https://mlsanity.com/paper/1802.05438/

## Key facts

- Cited by 207 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2018.

## Built on by (most cited first, 30 of 207)

- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.07489/) (2022)
- [Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing](https://mlsanity.com/paper/2102.07475/) (2021)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.09870/) (2023)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint](https://mlsanity.com/paper/2106.03787/) (2021)
- [Learning Multi-Agent Communication from Graph Modeling Perspective](https://mlsanity.com/paper/2405.08550/) (2024)
- [Settling the Variance of Multi-Agent Policy Gradients](https://mlsanity.com/paper/2108.08612/) (2021)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00285/) (2021)
- [Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2107.04050/) (2021)
- [Robust Multi-Agent Reinforcement Learning with State Uncertainty](https://mlsanity.com/paper/2307.16212/) (2023)
- [Reinforcement Learning for Ridesharing: An Extended Survey](https://mlsanity.com/paper/2105.01099/) (2021)
- [Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey](https://mlsanity.com/paper/2110.13484/) (2021)
- [Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization](https://mlsanity.com/paper/2110.13827/) (2021)
- [Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms](https://mlsanity.com/paper/2310.10810/) (2023)
- [Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts](https://mlsanity.com/paper/2105.03363/) (2021)
- [Solving Continuous Control via Q-learning](https://mlsanity.com/paper/2210.12566/) (2022)
- [A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising](https://mlsanity.com/paper/2106.06224/) (2021)
- [Global Convergence of Localized Policy Iteration in Networked Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2211.17116/) (2022)
- [On the Complexity of Computing Markov Perfect Equilibrium in General-Sum Stochastic Games](https://mlsanity.com/paper/2109.01795/) (2021)
- [A Survey on Reinforcement Learning in Aviation Applications](https://mlsanity.com/paper/2211.02147/) (2022)
- [Maximum Entropy Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.10715/) (2023)
- [Online Double Oracle](https://mlsanity.com/paper/2103.07780/) (2021)
- [Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.03319/) (2021)
- [Influencing Long-Term Behavior in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2203.03535/) (2022)
- [ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxation](https://mlsanity.com/paper/2109.06795/) (2021)
- [Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Training](https://mlsanity.com/paper/2107.14316/) (2021)
- [An Introduction to Multi-Agent Reinforcement Learning and Review of its Application to Autonomous Mobility](https://mlsanity.com/paper/2203.07676/) (2022)

## Abstract (arXiv)

Existing multi-agent reinforcement learning methods are limited typically to a small number of agents. When the agent number increases largely, the learning becomes intractable due to the curse of the dimensionality and the exponential growth of agent interactions. In this paper, we present Mean Field Reinforcement Learning where the interactions within the population of agents are approximated by those between a single agent and the average effect from the overall population or neighboring agents; the interplay between the two entities is mutually reinforced: the learning of the individual agent's optimal policy depends on the dynamics of the population, while the dynamics of the population change according to the collective patterns of the individual policies. We develop practical mean field Q-learning and mean field Actor-Critic algorithms and analyze the convergence of the solution to Nash equilibrium. Experiments on Gaussian squeeze, Ising model, and battle games justify the learning effectiveness of our mean field approaches. In addition, we report the first result to solve the Ising model via model-free reinforcement learning methods.