# Actor critic learning algorithms for mean-field control with moment neural networks

arXiv: https://arxiv.org/abs/2309.04317 · HTML: https://mlsanity.com/paper/2309.04317/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 8 Sep 2023.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems](https://mlsanity.com/paper/1912.01825/) (2019)
- [Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning](https://mlsanity.com/paper/1910.12802/) (2019)
- [Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis](https://mlsanity.com/paper/2002.04131/) (2020)
- [Unified Reinforcement Q-Learning for Mean Field Game and Control Problems](https://mlsanity.com/paper/2006.13912/) (2020)
- [Actor-Critic learning for mean-field control in continuous time](https://mlsanity.com/paper/2303.06993/) (2023)
- [Numerical resolution of McKean-Vlasov FBSDEs using neural networks](https://mlsanity.com/paper/1909.12678/) (2019)
- [Mean-field neural networks: learning mappings on Wasserstein space](https://mlsanity.com/paper/2210.15179/) (2022)
- [Learning High-Dimensional McKean-Vlasov Forward-Backward Stochastic Differential Equations with General Distribution Dependence](https://mlsanity.com/paper/2204.11924/) (2022)
- [DeepSets and their derivative networks for solving symmetric PDEs](https://mlsanity.com/paper/2103.00838/) (2021)
- [A fast iterative PDE-based algorithm for feedback controls of nonsmooth mean-field control problems](https://mlsanity.com/paper/2108.06740/) (2021)
- [Mean-field neural networks-based algorithms for McKean-Vlasov control problems *](https://mlsanity.com/paper/2212.11518/) (2022)
- [Quantile and moment neural networks for learning functionals of distributions](https://mlsanity.com/paper/2303.11060/) (2023)

## Built on by (most cited first, 7 of 7)

- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching](https://mlsanity.com/paper/2404.17939/) (2024)
- [Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise](https://mlsanity.com/paper/2408.02489/) (2024)
- [Model-free policy gradient for discrete-time mean-field control](https://mlsanity.com/paper/2601.11217/) (2026)
- [Policy Gradient for Continuous-Time Mean-Field Control](https://mlsanity.com/paper/2605.20718/) (2026)
- [Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data](https://mlsanity.com/paper/2606.26498/) (2026)
- [Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies](https://mlsanity.com/paper/2607.11005/) (2026)

## Abstract (arXiv)

We develop a new policy gradient and actor-critic algorithm for solving mean-field control problems within a continuous time reinforcement learning setting. Our approach leverages a gradient-based representation of the value function, employing parametrized randomized policies. The learning for both the actor (policy) and critic (value function) is facilitated by a class of moment neural network functions on the Wasserstein space of probability measures, and the key feature is to sample directly trajectories of distributions. A central challenge addressed in this study pertains to the computational treatment of an operator specific to the mean-field framework. To illustrate the effectiveness of our methods, we provide a comprehensive set of numerical results. These encompass diverse examples, including multi-dimensional settings and nonlinear quadratic mean-field control problems with controlled volatility.