# Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learning

arXiv: https://arxiv.org/abs/2006.11917 · HTML: https://mlsanity.com/paper/2006.11917/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2020.

## Built on by (most cited first, 21 of 21)

- [Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2107.04050/) (2021)
- [Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms](https://mlsanity.com/paper/2310.10810/) (2023)
- [Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach](https://mlsanity.com/paper/2105.08268/) (2021)
- [A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.03859/) (2022)
- [Minimax Optimal Kernel Operator Learning via Multilevel Training](https://mlsanity.com/paper/2209.14430/) (2022)
- [Dealing With Non-stationarity in Decentralized Cooperative Multi-Agent Deep Reinforcement Learning via Multi-Timescale Learning](https://mlsanity.com/paper/2302.02792/) (2023)
- [Relational Reasoning via Set Transformers: Provable Efficiency and Applications to MARL](https://mlsanity.com/paper/2209.09845/) (2022)
- [Transformers in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.05979/) (2023)
- [Indexability is Not Enough for Whittle: Improved, Near-Optimal Algorithms for Restless Bandits](https://mlsanity.com/paper/2211.00112/) (2022)
- [Learning Regularized Monotone Graphon Mean-Field Games](https://mlsanity.com/paper/2310.08089/) (2023)
- [When is Mean-Field Reinforcement Learning Tractable and Relevant?](https://mlsanity.com/paper/2402.05757/) (2024)
- [LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems](https://mlsanity.com/paper/2511.19368/) (2025)
- [Graphon Mean Field Games with a Representative Player: Analysis and Learning Algorithm](https://mlsanity.com/paper/2405.08005/) (2024)
- [Safe Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2306.17052/) (2023)
- [Population-size-Aware Policy Optimization for Mean-Field Games](https://mlsanity.com/paper/2302.03364/) (2023)
- [Exploiting Approximate Symmetry for Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2408.15173/) (2024)
- [Networked Communication for Decentralised Agents in Mean-Field Games](https://mlsanity.com/paper/2306.02766/) (2023)
- [Amortized Network Intervention to Steer the Excitatory Point Processes](https://mlsanity.com/paper/2310.04159/) (2023)
- [Explicit Credit Assignment through Local Rewards and Dependence Graphs in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2601.21523/) (2026)
- [Decentralized Strategies for Finite Population LQG Social Control: A Reinforcement Learning Approach](https://mlsanity.com/paper/2608.30244/) (2026)
- [Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations](https://mlsanity.com/paper/2609.02928/) (2026)

## Abstract (arXiv)

Multi-agent reinforcement learning (MARL) achieves significant empirical successes. However, MARL suffers from the curse of many agents. In this paper, we exploit the symmetry of agents in MARL. In the most generic form, we study a mean-field MARL problem. Such a mean-field MARL is defined on mean-field states, which are distributions that are supported on continuous space. Based on the mean embedding of the distributions, we propose MF-FQI algorithm that solves the mean-field MARL and establishes a non-asymptotic analysis for MF-FQI algorithm. We highlight that MF-FQI algorithm enjoys a "blessing of many agents" property in the sense that a larger number of observed agents improves the performance of MF-FQI algorithm.