# Mean Field Markov Decision Processes

arXiv: https://arxiv.org/abs/2106.08755 · HTML: https://mlsanity.com/paper/2106.08755/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2021.

## Built on by (most cited first, 10 of 10)

- [Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2107.04050/) (2021)
- [Major-Minor Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.10665/) (2023)
- [Safe Model-Based Multi-Agent Mean-Field Reinforcement Learning](https://mlsanity.com/paper/2306.17052/) (2023)
- [Model-free policy gradient for discrete-time mean-field control](https://mlsanity.com/paper/2601.11217/) (2026)
- [An Optimal Control Approach To Transformer Training](https://mlsanity.com/paper/2603.09571/) (2026)
- [Learning to Stop: Deep Learning for Mean Field Optimal Stopping](https://mlsanity.com/paper/2410.08850/) (2024)
- [Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data](https://mlsanity.com/paper/2606.26498/) (2026)
- [Scalable Ride-Sourcing Vehicle Rebalancing with Service Accessibility Guarantee: A Constrained Mean-Field Reinforcement Learning Approach](https://mlsanity.com/paper/2503.24183/) (2025)
- [QPLEX Decision Processes: Formulation via Nonlinear Markov Chains and Optimization via Policy Gradients](https://mlsanity.com/paper/2605.17149/) (2026)
- [Fair Policy Optimization in Major-Minor Weakly Coupled Markov Decision Processes](https://mlsanity.com/paper/2609.36174/) (2026)

## Abstract (arXiv)

We consider mean-field control problems in discrete time with discounted reward, infinite time horizon and compact state and action space. The existence of optimal policies is shown and the limiting mean-field problem is derived when the number of individuals tends to infinity. Moreover, we consider the average reward problem and show that the optimal policy in this mean-field limit is varepsilon-optimal for the discounted problem if the number of individuals is large and the discount factor close to one. This result is very helpful, because it turns out that in the special case when the reward does only depend on the distribution of the individuals, we obtain a very interesting subclass of problems where an average reward optimal policy can be obtained by first computing an optimal measure from a static optimization problem and then achieving it with Markov Chain Monte Carlo methods. We give two applications: Avoiding congestion an a graph and optimal positioning on a market place which we solve explicitly.