# Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spaces

arXiv: https://arxiv.org/abs/2309.10953 · HTML: https://mlsanity.com/paper/2309.10953/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning](https://mlsanity.com/paper/1910.12802/) (2019)
- [Entropy Regularization for Mean Field Games with Learning](https://mlsanity.com/paper/2010.00145/) (2020)
- [Unified Reinforcement Q-Learning for Mean Field Game and Control Problems](https://mlsanity.com/paper/2006.13912/) (2020)
- [Scalable Deep Reinforcement Learning Algorithms for Mean Field Games](https://mlsanity.com/paper/2203.11973/) (2022)
- [Mean Field Games Flock! The Reinforcement Learning Way](https://mlsanity.com/paper/2105.07933/) (2021)
- [Actor-Critic learning for mean-field control in continuous time](https://mlsanity.com/paper/2303.06993/) (2023)
- [Deep Learning for Mean Field Games and Mean Field Control with Applications to Finance](https://mlsanity.com/paper/2107.04568/) (2021)
- [Reinforcement Learning for Mean Field Games, with Applications to Economics](https://mlsanity.com/paper/2106.13755/) (2021)
- [Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulator](https://mlsanity.com/paper/2208.08744/) (2022)
- [Reinforcement Learning Algorithm for Mixed Mean Field Control Games](https://mlsanity.com/paper/2205.02330/) (2022)
- [Reinforcement Learning for Intra-and-Inter-Bank Borrowing and Lending Mean Field Control Game](https://mlsanity.com/paper/2207.03449/) (2022)

## Built on by (most cited first, 6 of 6)

- [Unified continuous-time q-learning for mean-field game and mean-field control problems](https://mlsanity.com/paper/2407.04521/) (2024)
- [Model-free policy gradient for discrete-time mean-field control](https://mlsanity.com/paper/2601.11217/) (2026)
- [Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms](https://mlsanity.com/paper/2604.27378/) (2026)
- [Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces](https://mlsanity.com/paper/2511.06812/) (2025)
- [Reinforcement Learning for Finite Space Mean-Field Type Games](https://mlsanity.com/paper/2409.18152/) (2024)
- [Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics](https://mlsanity.com/paper/2510.22158/) (2025)

## Abstract (arXiv)

We present the development and analysis of a reinforcement learning (RL) algorithm designed to solve continuous-space mean field game (MFG) and mean field control (MFC) problems in a unified manner. The proposed approach pairs the actor-critic (AC) paradigm with a representation of the mean field distribution via a parameterized score function, which can be efficiently updated in an online fashion, and uses Langevin dynamics to obtain samples from the resulting distribution. The AC agent and the score function are updated iteratively to converge, either to the MFG equilibrium or the MFC optimum for a given mean field problem, depending on the choice of learning rates. A straightforward modification of the algorithm allows us to solve mixed mean field control games (MFCGs). The performance of our algorithm is evaluated using linear-quadratic benchmarks in the asymptotic infinite horizon framework.