# Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time

arXiv: https://arxiv.org/abs/2008.06845 · HTML: https://mlsanity.com/paper/2008.06845/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2020.

## Built on by (most cited first, 20 of 20)

- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence](https://mlsanity.com/paper/2105.11066/) (2021)
- [Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2107.04050/) (2021)
- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping](https://mlsanity.com/paper/2312.12065/) (2023)
- [Reinforcement Learning for Mean Field Games, with Applications to Economics](https://mlsanity.com/paper/2106.13755/) (2021)
- [Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2407.17226/) (2024)
- [A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee](https://mlsanity.com/paper/2302.05816/) (2023)
- [Learning Decentralized Partially Observable Mean Field Control for Artificial Collective Behavior](https://mlsanity.com/paper/2307.06175/) (2023)
- [Towards a Theoretical Foundation of Policy Optimization for Learning Control Policies](https://mlsanity.com/paper/2210.04810/) (2022)
- [Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspective](https://mlsanity.com/paper/2110.13799/) (2021)
- [Safe Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2306.17052/) (2023)
- [Model-free policy gradient for discrete-time mean-field control](https://mlsanity.com/paper/2601.11217/) (2026)
- [Policy Gradient for Continuous-Time Mean-Field Control](https://mlsanity.com/paper/2605.20718/) (2026)
- [Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients](https://mlsanity.com/paper/2505.18379/) (2025)
- [Policy Optimization for Continuous-time Linear-Quadratic Graphon Mean Field Games](https://mlsanity.com/paper/2506.05894/) (2025)
- [Why does the two-timescale Q-learning converge to different mean field solutions? A unified convergence analysis](https://mlsanity.com/paper/2404.04357/) (2024)
- [Convergence of Policy Gradient for Stochastic Linear-Quadratic Control Problem in Infinite Horizon](https://mlsanity.com/paper/2404.11382/) (2024)
- [Fast Stochastic Policy Gradient: Negative Momentum for Reinforcement Learning](https://mlsanity.com/paper/2405.12228/) (2024)
- [Second-Order Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/2511.02095/) (2025)

## Abstract (arXiv)

Reinforcement learning is a powerful tool to learn the optimal policy of possibly multiple agents by interacting with the environment. As the number of agents grow to be very large, the system can be approximated by a mean-field problem. Therefore, it has motivated new research directions for mean-field control (MFC) and mean-field game (MFG). In this paper, we study the policy gradient method for the linear-quadratic mean-field control and game, where we assume each agent has identical linear state transitions and quadratic cost functions. While most of the recent works on policy gradient for MFC and MFG are based on discrete-time models, we focus on the continuous-time models where some analyzing techniques can be interesting to the readers. For both MFC and MFG, we provide policy gradient update and show that it converges to the optimal solution at a linear rate, which is verified by a synthetic simulation. For MFG, we also provide sufficient conditions for the existence and uniqueness of the Nash equilibrium.