mlsanityData to 7 Oct 2026

Generalization in Mean Field Games by Learning Master Policies

arXiv 2109.09717 ↗First posted 20 Sep 2021cs.LG cs.GT cs.MAcited by 28

Not on the map. Why?

Abstract (arXiv)

Mean Field Games (MFGs) can potentially scale multi-agent systems to extremely large populations of agents. Yet, most of the literature assumes a single initial distribution for the agents, which limits the practical applications of MFGs. Machine Learning has the potential to solve a wider diversity of MFG problems thanks to generalizations capacities. We study how to leverage these generalization properties to learn policies enabling a typical agent to behave optimally against any population distribution. In reference to the Master equation in MFGs, we coin the term ``Master policies'' to describe them and we prove that a single Master policy provides a Nash equilibrium, whatever the initial distribution. We propose a method to learn such Master policies. Our approach relies on three ingredients: adding the current population distribution as part of the observation, approximating Master policies with neural networks, and training via Reinforcement Learning and Fictitious Play. We illustrate on numerical examples not only the efficiency of the learned Master policy but also its generalization capabilities beyond the distributions used for training.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 20 Sep 2021.

Builds on

31 of 53 references.

References · 31

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2015Variational Inference with Normalizing Flowscited by 2,161
  6. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  7. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
  8. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 65
  9. 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 57
  10. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 56
  11. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
  12. 2017RLlib: Abstractions for Distributed Reinforcement Learningcited by 53
19 more
  1. 2020Entropy Regularization for Mean Field Games with Learningcited by 50
  2. 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 45
  3. 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 44
  4. 2020Q-Learning in Regularized Mean-field Gamescited by 40
  5. 2015Learning in Mean Field Games: the Fictitious Playcited by 37
  6. 2019Convergence Analysis of Machine Learning Algorithms for the Numerical Solution of Mean Field Control and Games: II -- The Finite Horizon Casecited by 34
  7. 2021Mean Field Games Flock! The Reinforcement Learning Waycited by 30
  8. 2021Scaling up Mean Field Games with Online Mirror Descentcited by 29
  9. 2019Mean-field Markov decision processes with common noise and open-loop controlscited by 24
  10. 2020A General Framework for Learning Mean-Field Gamescited by 24
  11. 2016Mean Field Game of Controls and An Application To Trade Crowdingcited by 22
  12. 2021Numerical Methods for Mean Field Games and Mean Field Type Controlcited by 21
  13. 2018Decentralised Learning in Systems with Many, Many Strategic Agentscited by 19
  14. 2019Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Gamescited by 15
  15. 2017Learning in anonymous nonatomic games with applications to first-order mean field gamescited by 10
  16. 2021DeepSets and their derivative networks for solving symmetric PDEscited by 10
  17. 2014Linear-Quadratic Mean Field Gamescited by 7
  18. 2020Model-free Reinforcement Learning for Non-stationary Mean Field Gamescited by 7
  19. 2014The Master Equation in Mean Field Theorycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 4; 2024: 9; 2025: 3; 2026: 4.

28 papers.

Sort:

By area:

  1. 2022Scalable Deep Reinforcement Learning Algorithms for Mean Field Gamescited by 33
  2. 2021Learning Graphon Mean Field Games and Approximate Nash Equilibriacited by 28
  3. 2023Recent Developments in Machine Learning Methods for Stochastic Control and Gamescited by 19
  4. 2022Learning Sparse Graphon Mean Field Gamescited by 14
  5. 2022A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learningcited by 12
  6. 2022Towards Multi-Agent Reinforcement Learning driven Over-The-Counter Market Simulationscited by 10
  7. 2024Unified continuous-time q-learning for mean-field game and mean-field control problemscited by 7
  8. 2024Deep Backward and Galerkin Methods for the Finite State Master Equationcited by 6
  9. 2024Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learningcited by 5
  10. 2024Graphon Mean Field Games with a Representative Player: Analysis and Learning Algorithmcited by 5
  11. 2023Major-Minor Mean Field Multi-Agent Reinforcement Learningcited by 4
  12. 2025Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learningcited by 4
16 more
  1. 2022Hypergraphon Mean Field Gamescited by 3
  2. 2023Population-size-Aware Policy Optimization for Mean-Field Gamescited by 3
  3. 2024Unsupervised Solution Operator Learning for Mean-Field Games via Sampling-Invariant Parametrizationscited by 3
  4. 2022Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environmentscited by 2
  5. 2024Exploiting Approximate Symmetry for Efficient Multi-Agent Reinforcement Learningcited by 2
  6. 2025Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Gamescited by 2
  7. 2026Operator Learning for Families of Finite-State Mean-Field Gamescited by 2
  8. 2022Scalable Task-Driven Robotic Swarm Control via Collision Avoidance and Learning Mean-Field Controlcited by 1
  9. 2024Reinforcement Learning for Finite Space Mean-Field Type Gamescited by 1
  10. 2023Networked Communication for Decentralised Agents in Mean-Field Gamescited by 0
  11. 2024Networked Communication for Mean-Field Games with Function Approximation and Empirical Mean-Field Estimationcited by 0
  12. 2024Last Iterate Convergence in Monotone Mean Field Gamescited by 0
  13. 2025PIONM: A Generalized Approach to Solving Density-Constrained Mean-Field Games Equilibrium under Modified Boundary Conditionscited by 0
  14. 2026Mean-Field Control on Sparse Graphs: From Local Limits to GNNs via Neighborhood Distributionscited by 0
  15. 2026Recurrent Structural Policy Gradient for Partially Observable Mean Field Gamescited by 0
  16. 2026Population-Aware Imitation Learning in Mean-field Games with Common Noisecited by 0