mlsanityData to 14 Oct 2026

Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games

arXiv 2505.22781 ↗First posted 28 May 2025stat.ML cs.LG math.STcited by 2

Not on the map. Why?

Abstract (arXiv)

We introduce Mean-Field Trust Region Policy Optimization (MF-TRPO), a novel algorithm designed to compute approximate Nash equilibria for ergodic Mean-Field Games (MFG) in finite state-action spaces. Building on the well-established performance of TRPO in the reinforcement learning (RL) setting, we extend its methodology to the MFG framework, leveraging its stability and robustness in policy optimization. Under standard assumptions in the MFG literature, we provide a rigorous analysis of MF-TRPO, establishing theoretical guarantees on its convergence. Our results cover both the exact formulation of the algorithm and its sample-based counterpart, where we derive high-probability guarantees and finite sample complexity. This work advances MFG optimization by bridging RL techniques with mean-field decision-making, offering a theoretically grounded approach to solving complex multi-agent problems.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 44 of its 95 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

44 of 95 references.

References · 44

  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  3. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  5. 2019The StarCraft Multi-Agent Challengecited by 597
  6. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  7. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  8. 2019A Theory of Regularized Markov Decision Processescited by 286
  9. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  10. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 261
  11. 2017Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equationscited by 191
  12. 2017A unified view of entropy-regularized Markov decision processescited by 182
32 more
  1. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  2. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  3. 2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 127
  4. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
  5. 2018Tight Analyses for Non-Smooth Stochastic Gradient Descentcited by 97
  6. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 91
  7. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 73
  8. 2020Fictitious Play for Mean Field Games: Continuous Time Analysis and Applicationscited by 62
  9. 2022Learning in Mean Field Games: A Surveycited by 61
  10. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
  11. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 49
  12. 2020Q-Learning in Regularized Mean-field Gamescited by 42
  13. 2019Convergence Analysis of Machine Learning Algorithms for the Numerical Solution of Mean Field Control and Games: II -- The Finite Horizon Casecited by 34
  14. 2021Mean Field Games Flock! The Reinforcement Learning Waycited by 31
  15. 2021Generalization in Mean Field Games by Learning Master Policiescited by 28
  16. 2021Learning Graphon Mean Field Games and Approximate Nash Equilibriacited by 28
  17. 2021Metrics and continuity in reinforcement learningcited by 26
  18. 2022Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Gamescited by 26
  19. 2020A General Framework for Learning Mean-Field Gamescited by 25
  20. 2020Mean Field Games and Applications: Numerical Aspectscited by 22
  21. 2022Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Pathcited by 20
  22. 2019Numerical resolution of McKean-Vlasov FBSDEs using neural networkscited by 19
  23. 2023Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problemscited by 10
  24. 2024Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learningcited by 9
  25. 2014Linear-Quadratic N-person and Mean-Field Games with Ergodic Costcited by 5
  26. 2023Regularization of the policy updates for stabilizing Mean Field Gamescited by 5
  27. 2015On Solutions of Mean Field Games with Ergodic Costcited by 2
  28. 2019Discrete-time average-cost mean-field games on Polish spacescited by 2
  29. 2024Common Noise by Random Measures: Constructing Mean-Field Equilibria for Competitive Investment and Hedgingcited by 2
  30. 2019On the asymptotic nature of first order mean field gamescited by 1
  31. 2021N-player games and mean field games of moderate interactionscited by 1
  32. 2023A mean-field game model of electricity market dynamicscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamicscited by 2
  2. 2026Towards Model-Free Learning in Dynamic Population Games: An Application to Karma Economiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024When is Mean-Field Reinforcement Learning Tractable and Relevant?
  2. 2026Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games
  3. 2024Reinforcement Learning for Finite Space Mean-Field Type Games
  4. 2025Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games
  5. 2024Exploiting Approximate Symmetry for Efficient Multi-Agent Reinforcement Learning
  6. 2024Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL
  7. 2024Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective
  8. 2024Efficient and Scalable Deep Reinforcement Learning for Mean Field Control Games