# Quantitative propagation of chaos for mean field Markov decision process with common noise

arXiv: https://arxiv.org/abs/2207.12738 · HTML: https://mlsanity.com/paper/2207.12738/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2022.

## Built on by (most cited first, 2 of 2)

- [Major-Minor Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.10665/) (2023)
- [Robust Q-learning for mean-field control under Wasserstein uncertainty in common noise](https://mlsanity.com/paper/2606.20356/) (2026)

## Abstract (arXiv)

We investigate propagation of chaos for mean field Markov Decision Process with common noise (CMKV-MDP), and when the optimization is performed over randomized open-loop controls on infinite horizon. We first state a rate of convergence of order MNγ, where MN is the mean rate of convergence in Wasserstein distance of the empirical measure, and γ ∈ (0,1] is an explicit constant, in the limit of the value functions of N-agent control problem with asymmetric open-loop controls, towards the value function of CMKV-MDP. Furthermore, we show how to explicitly construct (ε+O(MNγ))-optimal policies for the N-agent model from ε-optimal policies for the CMKV-MDP. Our approach relies on sharp comparison between the Bellman operators in the N-agent problem and the CMKV-MDP, and fine coupling of empirical measures.