# Actor-Critic learning for mean-field control in continuous time

arXiv: https://arxiv.org/abs/2303.06993 · HTML: https://mlsanity.com/paper/2303.06993/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 19 of 19)

- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning](https://mlsanity.com/paper/1910.12802/) (2019)
- [Entropy Regularization for Mean Field Games with Learning](https://mlsanity.com/paper/2010.00145/) (2020)
- [Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis](https://mlsanity.com/paper/2002.04131/) (2020)
- [Unified Reinforcement Q-Learning for Mean Field Game and Control Problems](https://mlsanity.com/paper/2006.13912/) (2020)
- [On the Convergence of Model Free Learning in Mean Field Games](https://mlsanity.com/paper/1907.02633/) (2019)
- [Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods](https://mlsanity.com/paper/1910.04295/) (2019)
- [Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach](https://mlsanity.com/paper/2108.02731/) (2021)
- [Numerical resolution of McKean-Vlasov FBSDEs using neural networks](https://mlsanity.com/paper/1909.12678/) (2019)
- [Reinforcement Learning for Mean Field Games, with Applications to Economics](https://mlsanity.com/paper/2106.13755/) (2021)
- [Learning High-Dimensional McKean-Vlasov Forward-Backward Stochastic Differential Equations with General Distribution Dependence](https://mlsanity.com/paper/2204.11924/) (2022)
- [Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics](https://mlsanity.com/paper/1604.04057/) (2016)
- [DeepSets and their derivative networks for solving symmetric PDEs](https://mlsanity.com/paper/2103.00838/) (2021)
- [A fast iterative PDE-based algorithm for feedback controls of nonsmooth mean-field control problems](https://mlsanity.com/paper/2108.06740/) (2021)
- [Mean-field neural networks-based algorithms for McKean-Vlasov control problems *](https://mlsanity.com/paper/2212.11518/) (2022)
- [Ito's formula for flows of measures on semimartingales](https://mlsanity.com/paper/2010.05288/) (2020)
- [A Probabilistic approach to classical solutions of the master equation for large population equilibria](https://mlsanity.com/paper/1411.3009/) (2014)

## Built on by (most cited first, 22 of 22)

- [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023)
- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2503.09981/) (2025)
- [Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2407.17226/) (2024)
- [Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty](https://mlsanity.com/paper/2404.12598/) (2024)
- [Actor critic learning algorithms for mean-field control with moment neural networks](https://mlsanity.com/paper/2309.04317/) (2023)
- [Score-based Neural Ordinary Differential Equations for Computing Mean Field Control Problems](https://mlsanity.com/paper/2409.16471/) (2024)
- [Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spaces](https://mlsanity.com/paper/2309.10953/) (2023)
- [Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise](https://mlsanity.com/paper/2408.02489/) (2024)
- [Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations](https://mlsanity.com/paper/2604.27372/) (2026)
- [Model-free policy gradient for discrete-time mean-field control](https://mlsanity.com/paper/2601.11217/) (2026)
- [Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms](https://mlsanity.com/paper/2604.27378/) (2026)
- [Policy Gradient for Continuous-Time Mean-Field Control](https://mlsanity.com/paper/2605.20718/) (2026)
- [Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games](https://mlsanity.com/paper/2405.17017/) (2024)
- [Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control](https://mlsanity.com/paper/2504.21793/) (2025)
- [Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces](https://mlsanity.com/paper/2511.06812/) (2025)
- [Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data](https://mlsanity.com/paper/2606.26498/) (2026)
- [Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies](https://mlsanity.com/paper/2607.11005/) (2026)
- [Two Kinds of Learning Algorithms for Continuous-Time VWAP Targeting Execution](https://mlsanity.com/paper/2411.06645/) (2024)
- [Neural Mean-Field Games: Extending Mean-Field Game Theory with Neural Stochastic Differential Equations](https://mlsanity.com/paper/2504.13228/) (2025)
- [Robust Q-learning for mean-field control under Wasserstein uncertainty in common noise](https://mlsanity.com/paper/2606.20356/) (2026)
- [Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning](https://mlsanity.com/paper/2607.03168/) (2026)

## Abstract (arXiv)

We study policy gradient for mean-field control in continuous time in a reinforcement learning setting. By considering randomised policies with entropy regularisation, we derive a gradient expectation representation of the value function, which is amenable to actor-critic type algorithms, where the value functions and the policies are learnt alternately based on observation samples of the state and model-free estimation of the population state distribution, either by offline or online learning. In the linear-quadratic mean-field framework, we obtain an exact parametrisation of the actor and critic functions defined on the Wasserstein space. Finally, we illustrate the results of our algorithms with some numerical experiments on concrete examples.