# Networked Communication for Mean-Field Games with Function Approximation and Empirical Mean-Field Estimation

arXiv: https://arxiv.org/abs/2408.11607 · HTML: https://mlsanity.com/paper/2408.11607/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Builds on (most cited first, 27 of 27)

- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Munchausen Reinforcement Learning](https://mlsanity.com/paper/2007.14430/) (2020)
- [Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications](https://mlsanity.com/paper/2007.03458/) (2020)
- [Learning in Mean Field Games: A Survey](https://mlsanity.com/paper/2205.12944/) (2022)
- [Q-Learning in Regularized Mean-field Games](https://mlsanity.com/paper/2003.12151/) (2020)
- [Scalable Deep Reinforcement Learning Algorithms for Mean Field Games](https://mlsanity.com/paper/2203.11973/) (2022)
- [Scaling up Mean Field Games with Online Mirror Descent](https://mlsanity.com/paper/2103.00623/) (2021)
- [Markov-Nash Equilibria in Mean-Field Games with Discounted Cost](https://mlsanity.com/paper/1612.07878/) (2016)
- [Mean Field Games Flock! The Reinforcement Learning Way](https://mlsanity.com/paper/2105.07933/) (2021)
- [Generalization in Mean Field Games by Learning Master Policies](https://mlsanity.com/paper/2109.09717/) (2021)
- [Multi Type Mean Field Reinforcement Learning](https://mlsanity.com/paper/2002.02513/) (2020)
- [Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Games](https://mlsanity.com/paper/2212.14449/) (2022)
- [A General Framework for Learning Mean-Field Games](https://mlsanity.com/paper/2003.06069/) (2020)
- [Numerical Methods for Mean Field Games and Mean Field Type Control](https://mlsanity.com/paper/2106.06231/) (2021)
- [Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Path](https://mlsanity.com/paper/2208.11639/) (2022)
- [Partially Observable Mean Field Reinforcement Learning](https://mlsanity.com/paper/2012.15791/) (2020)
- [Decentralized Mean Field Games](https://mlsanity.com/paper/2112.09099/) (2021)
- [Learning in anonymous nonatomic games with applications to first-order mean field games](https://mlsanity.com/paper/1704.00378/) (2017)
- [When is Mean-Field Reinforcement Learning Tractable and Relevant?](https://mlsanity.com/paper/2402.05757/) (2024)
- [Regularization of the policy updates for stabilizing Mean Field Games](https://mlsanity.com/paper/2304.01547/) (2023)
- [Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning](https://mlsanity.com/paper/2403.03552/) (2024)
- [MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games](https://mlsanity.com/paper/2405.00282/) (2024)
- [Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximation](https://mlsanity.com/paper/2408.08192/) (2024)
- [Mean-Field Games With Finitely Many Players: Independent Learning and Subjectivity](https://mlsanity.com/paper/2209.05703/) (2022)
- [Common Noise by Random Measures: Constructing Mean-Field Equilibria for Competitive Investment and Hedging](https://mlsanity.com/paper/2408.01175/) (2024)

## Abstract (arXiv)

Recent algorithms allow decentralised agents, possibly connected via a communication network, to learn equilibria in mean-field games from a non-episodic run of the empirical system. However, these algorithms are for tabular settings: this computationally limits the size of agents' observation space, meaning the algorithms cannot handle anything but small state spaces, nor generalise beyond policies depending only on the agent's local state to so-called 'population-dependent' policies. We address this limitation by introducing function approximation to the existing setting, drawing on the Munchausen Online Mirror Descent method that has previously been employed only in finite-horizon, episodic, centralised settings. While this permits us to include the mean field in the observation for players' policies, it is unrealistic to assume decentralised agents have access to this global information: we therefore also provide new algorithms allowing agents to locally estimate the global empirical distribution, and to improve this estimate via inter-agent communication. We prove theoretically that exchanging policy information helps networked agents outperform both independent and even centralised agents in function-approximation settings. Our experiments demonstrate this happening empirically, and show that the communication network allows decentralised agents to estimate the mean field for population-dependent policies.