mlsanityData to 7 Oct 2026

Regularization of the policy updates for stabilizing Mean Field Games

arXiv 2304.01547 ↗First posted 4 Apr 2023cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

This work studies non-cooperative Multi-Agent Reinforcement Learning (MARL) where multiple agents interact in the same environment and whose goal is to maximize the individual returns. Challenges arise when scaling up the number of agents due to the resultant non-stationarity that the many agents introduce. In order to address this issue, Mean Field Games (MFG) rely on the symmetry and homogeneity assumptions to approximate games with very large populations. Recently, deep Reinforcement Learning has been used to scale MFG to games with larger number of states. Current methods rely on smoothing techniques such as averaging the q-values or the updates on the mean-field distribution. This work presents a different approach to stabilize the learning based on proximal updates on the mean-field policy. We name our algorithm Mean Field Proximal Policy Optimization (MF-PPO), and we empirically show the effectiveness of our method in the OpenSpiel framework.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

20 of 33 references.

References · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  3. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  4. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  5. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 180
  6. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 153
  7. 2020Munchausen Reinforcement Learningcited by 68
  8. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 65
  9. 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 57
  10. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 42
  11. 2015Learning in Mean Field Games: the Fictitious Playcited by 37
  12. 2022Scalable Deep Reinforcement Learning Algorithms for Mean Field Gamescited by 33
8 more
  1. 2021Scaling up Mean Field Games with Online Mirror Descentcited by 29
  2. 2021Numerical Methods for Mean Field Games and Mean Field Type Controlcited by 21
  3. 2020Partially Observable Mean Field Reinforcement Learningcited by 16
  4. 2022Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Pathcited by 16
  5. 2019Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Gamescited by 15
  6. 2021Reinforcement Learning for Mean Field Games, with Applications to Economicscited by 14
  7. 2020Reinforcement Learning in Non-Stationary Discrete-Time Linear-Quadratic Mean-Field Gamescited by 8
  8. 2020Model-free Reinforcement Learning for Non-stationary Mean Field Gamescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 2.

5 papers.

  1. 2025Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Gamescited by 2
  2. 2026Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Gamescited by 1
  3. 2023Networked Communication for Decentralised Agents in Mean-Field Gamescited by 0
  4. 2024Networked Communication for Mean-Field Games with Function Approximation and Empirical Mean-Field Estimationcited by 0
  5. 2026Recurrent Structural Policy Gradient for Partially Observable Mean Field Gamescited by 0