# Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control

arXiv: https://arxiv.org/abs/2610.11619 · HTML: https://mlsanity.com/paper/2610.11619/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 9 of its 28 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods](https://mlsanity.com/paper/1910.04295/) (2019)
- [Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise](https://mlsanity.com/paper/2408.02489/) (2024)
- [Model-free policy gradient for discrete-time mean-field control](https://mlsanity.com/paper/2601.11217/) (2026)
- [Policy Gradient for Continuous-Time Mean-Field Control](https://mlsanity.com/paper/2605.20718/) (2026)
- [Learning operators on labelled conditional distributions with applications to mean field control of non exchangeable systems](https://mlsanity.com/paper/2603.21683/) (2026)
- [Probabilistic Analysis of Graphon Mean Field Control](https://mlsanity.com/paper/2505.19664/) (2025)
- [Non-Exchangeable Mean Field Markov Decision Processes with common noise : from Bellman equation to quantitative propagation of chaos](https://mlsanity.com/paper/2603.00009/) (2026)
- [A Zeroth-Order Deep Learning Method for Fully Nonlinear Parabolic Partial Differential Equations with Unknown Coefficients](https://mlsanity.com/paper/2606.24999/) (2026)

## Abstract (arXiv)

We develop a model-free policy gradient method for discrete-time mean-field control (MFC). In MFC, the policy affects the objective both through the controlled dynamics and through the population distribution. Standard REINFORCE estimators capture the first effect but not the second. We introduce Transport REINFORCE, a transport map-based approach that perturbs a suitable transformation of the population distribution to estimate this missing mean-field contribution. The method applies to both finite and continuous state spaces. In finite state spaces, we perturb the population distribution directly on the probability simplex through a convex combination of the current population weights and random weights. In continuous state spaces, we project the population distribution onto the manifold of Gaussian mixtures, and then randomize it via a transport map that ensures the perturbed law remains within this manifold. We prove consistency of the perturbed objective and gradient as the perturbation vanishes, and derive bias and mean-square error bounds for the resulting sample-based gradient estimator. Numerical experiments on several MFC benchmarks show that Transport REINFORCE improves over standard REINFORCE.