mlsanityData to 14 Oct 2026

Model-free policy gradient for discrete-time mean-field control

arXiv 2601.11217 ↗First posted 16 Jan 2026math.OC cs.LGcited by 4

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

We study model-free policy learning for discrete-time mean-field control (MFC) problems with finite state space and compact action space. In contrast to the extensive literature on value-based methods for MFC, policy-based approaches remain largely unexplored due to the intrinsic dependence of transition kernels and rewards on the evolving population state distribution, which prevents the direct use of likelihood-ratio estimators of policy gradients from classical single-agent reinforcement learning. We introduce a novel perturbation scheme on the state-distribution flow and prove that the gradient of the resulting perturbed value function converges to the true policy gradient as the perturbation magnitude vanishes. This construction yields a fully model-free estimator based solely on simulated trajectories and an auxiliary estimate of the sensitivity of the state distribution. Building on this framework, we develop MF-REINFORCE, a model-free policy gradient algorithm for MFC, and establish explicit quantitative bounds on its bias and mean-squared error. Numerical experiments on representative mean-field control tasks demonstrate the effectiveness of the proposed approach.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 35 of its 46 references that are in the map. First posted on arXiv 16 Jan 2026.

Builds on

35 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces10 Nov
  2. 2024Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise5 Aug

Other references · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  5. 2019A Theory of Regularized Markov Decision Processescited by 286
  6. 2018Mean Field Multi-Agent Reinforcement Learningcited by 207
  7. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  8. 2021Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learningcited by 73
  9. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 59
  10. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  11. 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 49
  12. 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 49
21 more
  1. 2020Q-Learning in Regularized Mean-field Gamescited by 42
  2. 2021On the Approximation of Cooperative Heterogeneous Multi-Agent Reinforcement Learning (MARL) using Mean Field Control (MFC)cited by 32
  3. 2023Sample Complexity of Variance-reduced Distributionally Robust Q-learningcited by 29
  4. 2019Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methodscited by 28
  5. 2021Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approachcited by 28
  6. 2019Deep Learning Methods for Mean Field Control Problems with Delaycited by 25
  7. 2019Mean-field Markov decision processes with common noise and open-loop controlscited by 25
  8. 2020A General Framework for Learning Mean-Field Gamescited by 25
  9. 2021Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Gamescited by 25
  10. 2019Dynamic Programming Principles for Mean-Field Controls with Learningcited by 23
  11. 2023Actor-Critic learning for mean-field control in continuous timecited by 22
  12. 2023Continuous-time q-learning for mean-field control problemscited by 22
  13. 2020Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Timecited by 20
  14. 2019Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Gamescited by 19
  15. 2022On the convergence of policy gradient methods to Nash equilibria in general stochastic gamescited by 16
  16. 2016Dynamic programming for optimal control of stochastic McKean-Vlasov dynamicscited by 12
  17. 2021Mean Field Markov Decision Processescited by 10
  18. 2015Mean-field-game model for Botnet defense in Cyber-securitycited by 8
  19. 2023Actor critic learning algorithms for mean-field control with moment neural networkscited by 7
  20. 2023Deep Reinforcement Learning for Infinite Horizon Mean Field Problems in Continuous Spacescited by 6
  21. 2025Learning algorithms for mean field optimal controlcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Datacited by 1
  2. 2026Actor-Critic Learning for Extended Mean Field Control with Deterministic Policiescited by 1
  3. 2026Mean Field Reinforcement Learningcited by 0
  4. 2026Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Policy Gradient for Continuous-Time Mean-Field Control
  2. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  3. 2024Efficient and Scalable Deep Reinforcement Learning for Mean Field Control Games
  4. 2026Model-Free Output Feedback Stabilization via Policy Gradient Methods
  5. 2026Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients
  6. 2024Unified continuous-time q-learning for mean-field game and mean-field control problems
  7. 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations
  8. 2025Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics