# Reinforcement Learning for Finite Space Mean-Field Type Games

arXiv: https://arxiv.org/abs/2409.18152 · HTML: https://mlsanity.com/paper/2409.18152/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 64 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 28 of 28)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Combining Deep Reinforcement Learning and Search for Imperfect-Information Games](https://mlsanity.com/paper/2007.13544/) (2020)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications](https://mlsanity.com/paper/2007.03458/) (2020)
- [Learning in Mean Field Games: A Survey](https://mlsanity.com/paper/2205.12944/) (2022)
- [Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning](https://mlsanity.com/paper/1910.12802/) (2019)
- [Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis](https://mlsanity.com/paper/2002.04131/) (2020)
- [On the Convergence of Model Free Learning in Mean Field Games](https://mlsanity.com/paper/1907.02633/) (2019)
- [Markov-Nash Equilibria in Mean-Field Games with Discounted Cost](https://mlsanity.com/paper/1612.07878/) (2016)
- [Generalization in Mean Field Games by Learning Master Policies](https://mlsanity.com/paper/2109.09717/) (2021)
- [Multi Type Mean Field Reinforcement Learning](https://mlsanity.com/paper/2002.02513/) (2020)
- [Mean-field Markov decision processes with common noise and open-loop controls](https://mlsanity.com/paper/1912.07883/) (2019)
- [Partially Observable Mean Field Reinforcement Learning](https://mlsanity.com/paper/2012.15791/) (2020)
- [Deep Q-Learning for Nash Equilibria: Nash-DQN](https://mlsanity.com/paper/1904.10554/) (2019)
- [Mean-Field-Type Games in Engineering](https://mlsanity.com/paper/1605.03281/) (2016)
- [Mean-field-game model for Botnet defense in Cyber-security](https://mlsanity.com/paper/1511.06642/) (2015)
- [Reinforcement Learning Algorithm for Mixed Mean Field Control Games](https://mlsanity.com/paper/2205.02330/) (2022)
- [Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior](https://mlsanity.com/paper/2307.06175/) (2023)
- [Mean Field Control and Mean Field Game Models with Several Populations](https://mlsanity.com/paper/1810.00783/) (2018)
- [Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spaces](https://mlsanity.com/paper/2309.10953/) (2023)
- [Policy Optimization for Linear-Quadratic Zero-Sum Mean-Field Type Games](https://mlsanity.com/paper/2009.02146/) (2020)
- [Robust Cooperative Multi-Agent Reinforcement Learning:A Mean-Field Type Game Perspective](https://mlsanity.com/paper/2406.13992/) (2024)
- [Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective](https://mlsanity.com/paper/2403.11345/) (2024)
- [Mean-field games among teams](https://mlsanity.com/paper/2310.12282/) (2023)
- [Risk-Sensitive Mean-Field-Type Games with Lp-norm Drifts](https://mlsanity.com/paper/1505.06280/) (2015)
- [Zero-sum stochastic differential games of generalized McKean-Vlasov type *](https://mlsanity.com/paper/1803.07329/) (2018)

## Built on by (most cited first, 2 of 2)

- [Robust Q-learning for mean-field control under Wasserstein uncertainty in common noise](https://mlsanity.com/paper/2606.20356/) (2026)
- [Beyond Nominal Equilibria: Risk-Averse Multi-Population Mean-Field Games](https://mlsanity.com/paper/2610.09244/) (2026)

## Abstract (arXiv)

Mean field type games (MFTGs) describe Nash equilibria between large coalitions: each coalition consists of a continuum of cooperative agents who maximize the average reward of their coalition while interacting non-cooperatively with a finite number of other coalitions. Although the theory has been extensively developed, we are still lacking efficient and scalable computational methods. Here, we develop reinforcement learning methods for such games in a finite space setting with general dynamics and reward functions. We start by proving that the MFTG solution yields approximate Nash equilibria in finite-size coalition games. We then propose two algorithms. The first is based on the quantization of mean-field spaces and Nash Q-learning. We provide convergence and stability analysis. We then propose a deep reinforcement learning algorithm, which can scale to larger spaces. Numerical experiments in 4 environments with mean-field distributions of dimension up to 200 show the scalability and efficiency of the proposed method.