mlsanityData to 14 Oct 2026

Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games

arXiv 2506.05894 ↗First posted 6 Jun 2025math.OC cs.LG math.PRcited by 1

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Multi-agent reinforcement learning, despite its popularity and empirical success, faces significant scalability challenges in large-population dynamic games. Graphon mean field games (GMFGs) offer a principled framework for approximating such games while capturing heterogeneity among players. In this paper, we propose and analyze a policy optimization framework for continuous-time, finite-horizon linear-quadratic GMFGs. Exploiting the structural properties of GMFGs, we design an efficient policy parameterization in which each player's policy is represented as an affine function of their private state, with a shared slope function and player-specific intercepts. We develop a bilevel optimization algorithm that alternates between policy gradient updates for best-response computation under a fixed population distribution, and distribution updates using the resulting policies. We prove linear convergence of the policy gradient steps to best-response policies and establish global convergence of the overall algorithm to the Nash equilibrium. The analysis relies on novel landscape characterizations over infinite-dimensional policy spaces. Numerical experiments demonstrate the convergence and robustness of the proposed algorithm under varying graphon structures, noise levels, and action frequencies.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 40 of its 63 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

40 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise5 Aug
  2. 2024MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games1 May
  3. 2024When is Mean-Field Reinforcement Learning Tractable and Relevant?8 Feb
  4. 2024Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL8 Feb

Other references · 36

  1. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  2. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  3. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  4. 2024Reinforcement learningcited by 247
  5. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 73
  6. 2019A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimizationcited by 68
  7. 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
  8. 2022Learning in Mean Field Games: A Surveycited by 61
  9. 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 50
  10. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 49
  11. 2019On the Convergence of Model Free Learning in Mean Field Gamescited by 48
  12. 2019Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methodscited by 28
24 more
  1. 2021Learning Graphon Mean Field Games and Approximate Nash Equilibriacited by 28
  2. 2022Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Gamescited by 26
  3. 2021Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Gamescited by 25
  4. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 25
  5. 2021Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methodscited by 23
  6. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
  7. 2020Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Timecited by 20
  8. 2021Deep Learning for Mean Field Games and Mean Field Control with Applications to Financecited by 20
  9. 2022Learning Sparse Graphon Mean Field Gamescited by 14
  10. 2023On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximationcited by 14
  11. 2019Stochastic Graphon Games: I. The Static Casecited by 13
  12. 2020Policy Gradient-based Algorithms for Continuous-time Linear Quadratic Controlcited by 13
  13. 2020Graphon Mean Field Games and the GMFG Equationscited by 13
  14. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 12
  15. 2023Learning Regularized Monotone Graphon Mean-Field Gamescited by 9
  16. 2019Value Iteration Algorithm for Mean-field Gamescited by 5
  17. 2020Linear Quadratic Graphon Field Gamescited by 5
  18. 2024Graphon Mean Field Games with a Representative Player: Analysis and Learning Algorithmcited by 5
  19. 2020LQG Graphon Mean Field Games: Analysis via Graphon Invariant Subspacescited by 4
  20. 2022A label-state formulation of stochastic graphon games and approximate equilibria on large networkscited by 4
  21. 2023Learning Regularized Graphon Mean-Field Games with Unknown Graphonscited by 4
  22. 2023Reinforcement Learning for SBM Graphon Games with Re-Samplingcited by 1
  23. 2024Linear-Quadratic Graphon Mean Field Games with Common Noisecited by 1
  24. 2024Stochastic Graphon Games with Memorycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Learning Distributed Equilibria in Linear-Quadratic Stochastic Differential Games: An α-Potential Approachcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games
  2. 2024Learning Mean Field Games on Sparse Graphs: A Hybrid Graphex Approach
  3. 2025Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games
  4. 2024Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective
  5. 2026Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learning
  6. 2025Learning in Stackelberg Mean Field Games: A Non-Asymptotic Analysis
  7. 2024Efficient and Scalable Deep Reinforcement Learning for Mean Field Control Games
  8. 2026Data-Driven Network LQG Mean Field Games with Heterogeneous Populations via Integral Reinforcement Learning