mlsanityData to 14 Oct 2026

Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learning

arXiv 2509.03030 ↗First posted 3 Sep 2025cs.LG cs.MA cs.ROcited by 4

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Mean Field Games (MFGs) offer a powerful framework for studying large-scale multi-agent systems. Yet, learning Nash equilibria in MFGs remains a challenging problem, particularly when the initial distribution is unknown or when the population is subject to common noise. In this paper, we introduce an efficient deep reinforcement learning (DRL) algorithm designed to achieve population-dependent Nash equilibria without relying on averaging or historical sampling, inspired by Munchausen RL and Online Mirror Descent. The resulting policy is adaptable to various initial distributions and sources of common noise. Through numerical experiments on seven canonical examples, we demonstrate that our algorithm exhibits superior convergence properties compared to state-of-the-art algorithms, particularly a DRL version of Fictitious Play for population-dependent policies. The performance in the presence of common noise underscores the robustness and adaptability of our approach.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 38 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

20 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning6 Mar

Other references · 19

  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  3. 2015Prioritized Experience Replaycited by 1,099
  4. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  5. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 73
  6. 2020Munchausen Reinforcement Learningcited by 69
  7. 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 62
  8. 2022Learning in Mean Field Games: A Surveycited by 61
  9. 2019Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learningcited by 57
  10. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  11. 2020Q-Learning in Regularized Mean-field Gamescited by 42
  12. 2015Learning in Mean Field Games: the Fictitious Playcited by 38
7 more
  1. 2021Training Larger Networks for Deep Reinforcement Learningcited by 38
  2. 2022Scalable Deep Reinforcement Learning Algorithms for Mean Field Gamescited by 38
  3. 2019Computing Approximate Equilibria in Sequential Adversarial Games by Exploitability Descentcited by 31
  4. 2021Generalization in Mean Field Games by Learning Master Policiescited by 28
  5. 2017Learning in anonymous nonatomic games with applications to first-order mean field gamescited by 11
  6. 2018Finite mean field games: fictitious play and convergence to a first order continuous mean field gamecited by 9
  7. 2014Linear-Quadratic Mean Field Gamescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Structural Reinforcement Learning for Heterogeneous Agent Macroeconomicscited by 2
  2. 2026Operator Learning for Families of Finite-State Mean-Field Gamescited by 2
  3. 2026Recurrent Structural Policy Gradient for Partially Observable Mean Field Gamescited by 0
  4. 2026Population-Aware Imitation Learning in Mean-field Games with Common Noisecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Scalable Offline Reinforcement Learning for Mean Field Games
  2. 2024MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games
  3. 2024Independent Policy Mirror Descent for Markov Potential Games: Scaling to Large Number of Players
  4. 2025Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics
  5. 2024Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective
  6. 2024Efficient and Scalable Deep Reinforcement Learning for Mean Field Control Games
  7. 2026DiffNash: Finding Nash Equilibrium in High-Dimensional Continuous Action Spaces with Diffusion Policies
  8. 2023On Imitation in Mean-field Games