mlsanityData to 7 Oct 2026

Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning

arXiv 2102.01585 ↗First posted 2 Feb 2021cs.MA cs.AI cs.LGcited by 65

Not on the map. Why?

Abstract (arXiv)

The recent mean field game (MFG) formalism facilitates otherwise intractable computation of approximate Nash equilibria in many-agent settings. In this paper, we consider discrete-time finite MFGs subject to finite-horizon objectives. We show that all discrete-time finite MFGs with non-constant fixed point operators fail to be contractive as typically assumed in existing MFG literature, barring convergence via fixed point iteration. Instead, we incorporate entropy-regularization and Boltzmann policies into the fixed point iteration. As a result, we obtain provable convergence to approximate fixed points where existing methods fail, and reach the original goal of approximate Nash equilibria. All proposed methods are evaluated with respect to their exploitability, on both instructive examples with tractable exact solutions and high-dimensional problems where exact methods become intractable. In high-dimensional scenarios, we apply established deep reinforcement learning methods and empirically combine fictitious play with our approximations.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 14 of its 35 references that are in the map. First posted on arXiv 2 Feb 2021.

Builds on

14 of 35 references.

References · 14

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  4. 2018Maximum a Posteriori Policy Optimisationcited by 318
  5. 2018Mean Field Multi-Agent Reinforcement Learningcited by 202
  6. 2017A unified view of entropy-regularized Markov decision processescited by 174
  7. 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 57
  8. 2020Q-Learning in Regularized Mean-field Gamescited by 40
  9. 2015Learning in Mean Field Games: the Fictitious Playcited by 37
  10. 2016Markov-Nash Equilibria in Mean-Field Games with Discounted Costcited by 29
  11. 2018Decentralised Learning in Systems with Many, Many Strategic Agentscited by 19
  12. 2019Entropic Regularization of Markov Decision Processescited by 15
2 more
  1. 2017Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactionscited by 8
  2. 2019Value Iteration Algorithm for Mean-field Gamescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 11; 2023: 12; 2024: 18; 2025: 11; 2026: 8.

65 papers.

Sort:

By area:

  1. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
  2. 2022Scalable Deep Reinforcement Learning Algorithms for Mean Field Gamescited by 33
  3. 2021Generalization in Mean Field Games by Learning Master Policiescited by 28
  4. 2021Learning Graphon Mean Field Games and Approximate Nash Equilibriacited by 28
  5. 2022Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Gamescited by 24
  6. 2023Recent Developments in Machine Learning Methods for Stochastic Control and Gamescited by 19
  7. 2022Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Pathcited by 16
  8. 2022Learning Sparse Graphon Mean Field Gamescited by 14
  9. 2023On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximationcited by 13
  10. 2022A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learningcited by 12
  11. 2022Individual-Level Inverse Reinforcement Learning for Mean Field Gamescited by 10
  12. 2023On Imitation in Mean-field Gamescited by 10
38 more
  1. 2024Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RLcited by 9
  2. 2021Adversarial Inverse Reinforcement Learning for Mean Field Gamescited by 8
  3. 2023Learning Regularized Monotone Graphon Mean-Field Gamescited by 8
  4. 2021Discrete-Time Mean Field Control with Environment Statescited by 7
  5. 2024Unified continuous-time q-learning for mean-field game and mean-field control problemscited by 7
  6. 2023Combining Tree-Search, Generative Models, and Nash Bargaining Concepts in Game-Theoretic Reinforcement Learningcited by 6
  7. 2023Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spacescited by 6
  8. 2023Population-based Evaluation in Repeated Rock-Paper-Scissors as a Benchmark for Multiagent Reinforcement Learningcited by 5
  9. 2023Regularization of the policy updates for stabilizing Mean Field Gamescited by 5
  10. 2024Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learningcited by 5
  11. 2024MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Gamescited by 5
  12. 2024Graphon Mean Field Games with a Representative Player: Analysis and Learning Algorithmcited by 5
  13. 2024Stochastic Semi-Gradient Descent for Learning Mean Field Games with Population-Aware Function Approximationcited by 5
  14. 2025Decision-Dependent Stochastic Optimization: The Role of Distribution Dynamicscited by 5
  15. 2023Major-Minor Mean Field Multi-Agent Reinforcement Learningcited by 4
  16. 2024Maximum Causal Entropy IRL in Mean-Field Games and GNEP Framework for Forward RLcited by 4
  17. 2025Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learningcited by 4
  18. 2022Hypergraphon Mean Field Gamescited by 3
  19. 2022Mean Field Games on Weighted and Directed Graphs via Colored Digraphonscited by 3
  20. 2023Population-size-Aware Policy Optimization for Mean-Field Gamescited by 3
  21. 2024Learning Mean Field Games on Sparse Graphs: A Hybrid Graphex Approachcited by 3
  22. 2024Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Gamescited by 3
  23. 2025Efficient Learning for Entropy-Regularized Markov Decision Processes via Multilevel Monte Carlocited by 3
  24. 2025Meta-Inverse Reinforcement Learning for Mean Field Games via Probabilistic Context Variablescited by 3
  25. 2025Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spacescited by 3
  26. 2026Model-free policy gradient for discrete-time mean-field controlcited by 3
  27. 2022Learning Mean-Field Control for Delayed Information Load Balancing in Large Queuing Systemscited by 2
  28. 2023Learning Discrete-Time Major-Minor Mean Field Gamescited by 2
  29. 2024Policy Optimization finds Nash Equilibrium in Regularized General-Sum LQ Gamescited by 2
  30. 2024A Single Online Agent Can Efficiently Learn Mean Field Gamescited by 2
  31. 2024Exploiting Approximate Symmetry for Efficient Multi-Agent Reinforcement Learningcited by 2
  32. 2025Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Gamescited by 2
  33. 2025Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamicscited by 2
  34. 2026Operator Learning for Families of Finite-State Mean-Field Gamescited by 2
  35. 2022Scalable Task-Driven Robotic Swarm Control via Collision Avoidance and Learning Mean-Field Controlcited by 1
  36. 2024Reinforcement Learning for Finite Space Mean-Field Type Gamescited by 1
  37. 2024Scalable Offline Reinforcement Learning for Mean Field Gamescited by 1
  38. 2025Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Gamescited by 1

50 most cited of 65 All 65 (JSON)