# Policy Gradient for Continuous-Time Mean-Field Control arXiv: https://arxiv.org/abs/2605.20718 · HTML: https://mlsanity.com/paper/2605.20718/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. - Builds on 42 of its 79 references that are in the map. - First posted on arXiv 20 May 2026. ## Placed in - Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/) ## Builds on (most cited first, 30 of 42) - [Provably Efficient Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/1907.05388/) (2019) - [Model-Based Reinforcement Learning with Value-Targeted Regression](https://mlsanity.com/paper/2006.01107/) (2020) - [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021) - [A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems](https://mlsanity.com/paper/1912.01825/) (2019) - [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022) - [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021) - [Making Deep Q-learning methods robust to time discretization](https://mlsanity.com/paper/1901.09732/) (2019) - [Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods](https://mlsanity.com/paper/1910.04295/) (2019) - [Efficient Local Planning with Linear Function Approximation](https://mlsanity.com/paper/2108.05533/) (2021) - [Numerical Methods for Mean Field Games and Mean Field Type Control](https://mlsanity.com/paper/2106.06231/) (2021) - [Actor-Critic learning for mean-field control in continuous time](https://mlsanity.com/paper/2303.06993/) (2023) - [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023) - [Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time](https://mlsanity.com/paper/2008.06845/) (2020) - [Mean-field neural networks: learning mappings on Wasserstein space](https://mlsanity.com/paper/2210.15179/) (2022) - [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024) - [Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2503.09981/) (2025) - [Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics](https://mlsanity.com/paper/1604.04057/) (2016) - [Limit theory for controlled McKean-Vlasov dynamics](https://mlsanity.com/paper/1609.08064/) (2016) - [Bellman equation and viscosity solutions for mean-field stochastic control problem](https://mlsanity.com/paper/1512.07866/) (2015) - [Actor critic learning algorithms for mean-field control with moment neural networks](https://mlsanity.com/paper/2309.04317/) (2023) - [Mean Field Type Control with Congestion (II): An Augmented Lagrangian Method](https://mlsanity.com/paper/1611.02023/) (2016) - [PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluation](https://mlsanity.com/paper/2405.12535/) (2024) - [Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise](https://mlsanity.com/paper/2408.02489/) (2024) - [Learning algorithms for mean field optimal control](https://mlsanity.com/paper/2503.17869/) (2025) - [Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms](https://mlsanity.com/paper/2604.27378/) (2026) - [Randomized dynamic programming principle and Feynman-Kac representation for optimal control of McKean-Vlasov dynamics](https://mlsanity.com/paper/1606.08204/) (2016) - [The Pontryagin Maximum Principle in the Wasserstein Space](https://mlsanity.com/paper/1711.07667/) (2017) - [McKean-Vlasov optimal control: limit theory and equivalence between different formulations](https://mlsanity.com/paper/2001.00925/) (2020) - [Propagation of Chaos of Forward-Backward Stochastic Differential Equations with Graphon Interactions](https://mlsanity.com/paper/2202.08163/) (2022) - [Comparison of viscosity solutions for a class of second order PDEs on the Wasserstein space](https://mlsanity.com/paper/2309.05040/) (2023) ## Built on by (most cited first, 4 of 4) - [Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data](https://mlsanity.com/paper/2606.26498/) (2026) - [Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies](https://mlsanity.com/paper/2607.11005/) (2026) - [Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence](https://mlsanity.com/paper/2609.14972/) (2026) - [Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control](https://mlsanity.com/paper/2610.11619/) (2026) ## Abstract (arXiv) This paper develops a policy gradient method for entropy-regularized mean-field control in the discounted infinite-horizon setting. We consider randomized feedback policies and a coupled representative-particle/population system, in which the representative state evolves jointly with a population law governed by a McKean--Vlasov equation. The resulting value function is therefore defined on the product space mathbb Rd × mathcal P2(mathbb Rd). A key distinction from existing policy gradient methods for mean-field control is that, after computing the value function under a fixed policy, our approach does not require solving an additional equation to obtain the policy gradient. Instead, we derive an explicit policy gradient formula directly in terms of the value function. The formulation is based on an instantaneous advantage function, which quantifies the gain of taking a given action relative to the current randomized policy. We establish a Gateaux policy-gradient formula, which gives the first-order variation of the objective along arbitrary policy perturbations, and then derive the corresponding ascent direction under finite-dimensional policy parametrization. The resulting formula leads to a model-based actor--critic scheme. The critic is obtained by solving the associated linear stationary Hamilton--Jacobi--Bellman equation for the value function, using cylindrical functions to represent dependence on the population law. The actor is then updated according to the derived policy-gradient formula. We further analyze the well-posedness of the PDE in a polynomial-growth function class. Finally, we illustrate the proposed method through numerical experiments on an LQR model and a crowd-motion problem.