# Decentralised Learning in Systems with Many, Many Strategic Agents

arXiv: https://arxiv.org/abs/1803.05028 · HTML: https://mlsanity.com/paper/1803.05028/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2018.

## Built on by (most cited first, 20 of 20)

- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergence](https://mlsanity.com/paper/2202.04129/) (2022)
- [Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint](https://mlsanity.com/paper/2106.03787/) (2021)
- [The Traitors: Deception and Trust in Multi-Agent Language Model Simulations](https://mlsanity.com/paper/2505.12923/) (2025)
- [Generalization in Mean Field Games by Learning Master Policies](https://mlsanity.com/paper/2109.09717/) (2021)
- [Learning Graphon Mean Field Games and Approximate Nash Equilibria](https://mlsanity.com/paper/2112.01280/) (2021)
- [Reinforcement Learning for Mean Field Games, with Applications to Economics](https://mlsanity.com/paper/2106.13755/) (2021)
- [A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.03859/) (2022)
- [Learning Sparse Graphon Mean Field Games](https://mlsanity.com/paper/2209.03880/) (2022)
- [Individual-Level Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2202.06401/) (2022)
- [Discrete-Time Mean Field Control with Environment States](https://mlsanity.com/paper/2104.14900/) (2021)
- [Equilibrium Inverse Reinforcement Learning for Ride-hailing Vehicle Network](https://mlsanity.com/paper/2102.06854/) (2021)
- [Population-size-Aware Policy Optimization for Mean-Field Games](https://mlsanity.com/paper/2302.03364/) (2023)
- [Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces](https://mlsanity.com/paper/2511.06812/) (2025)
- [A Single Online Agent Can Efficiently Learn Mean Field Games](https://mlsanity.com/paper/2405.03718/) (2024)
- [Taming Multi-Agent Reinforcement Learning with Estimator Variance Reduction](https://mlsanity.com/paper/2209.01054/) (2022)
- [Networked Communication for Decentralised Agents in Mean-Field Games](https://mlsanity.com/paper/2306.02766/) (2023)
- [Why does the two-timescale Q-learning converge to different mean field solutions? A unified convergence analysis](https://mlsanity.com/paper/2404.04357/) (2024)
- [Tacit Learning with Adaptive Information Selection for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.15639/) (2024)
- [Stationary Robust Mean-Field Games under Model Mismatches](https://mlsanity.com/paper/2606.22579/) (2026)

## Abstract (arXiv)

Although multi-agent reinforcement learning can tackle systems of strategically interacting entities, it currently fails in scalability and lacks rigorous convergence guarantees. Crucially, learning in multi-agent systems can become intractable due to the explosion in the size of the state-action space as the number of agents increases. In this paper, we propose a method for computing closed-loop optimal policies in multi-agent systems that scales independently of the number of agents. This allows us to show, for the first time, successful convergence to optimal behaviour in systems with an unbounded number of interacting adaptive learners. Studying the asymptotic regime of N-player stochastic games, we devise a learning protocol that is guaranteed to converge to equilibrium policies even when the number of agents is extremely large. Our method is model-free and completely decentralised so that each agent need only observe its local state information and its realised rewards. We validate these theoretical results by showing convergence to Nash-equilibrium policies in applications from economics and control theory with thousands of strategically interacting agents.