Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data
Not on the map. Why?
Abstract (arXiv)
This paper develops a model-free framework for continuous-time mean-field control when the population evolves according to unknown controlled McKean--Vlasov dynamics and only discrete-time transition data are available. Model-based mean-field control requires the continuous-time drift and diffusion coefficients, which are not directly observed from fixed-step transitions, while a direct reduction to a discrete-time Bellman equation loses the continuous-time generator structure. To bridge these two viewpoints, we introduce a Mean-Field-PhiBE (MF-PhiBE), which incorporates discrete-time transition information into a continuous-time PDE on the Wasserstein space. The MF-PhiBE replaces the unknown infinitesimal drift and covariance in the policy-evaluation equation by one-step estimators computed from data, while preserving the generator structure of the McKean-Vlasov HJB equation. We also derive a policy-gradient theorem for entropy-regularized randomized feedback policies, expressing the actor direction through an action-wise infinitesimal advantage and the score of the policy. Combining these two ingredients yields a model-free actor-critic method. We prove a first-order consistency estimate showing that the value induced by an optimal MF-PhiBE policy approximates the optimal continuous-time value as the observation time step vanishes. For entropy-regularized LQR, we establish first-order policy convergence and second-order value convergence; under suitable conditions, the population-averaged feedback means coincide exactly. Numerical experiments on an LQR benchmark and a crowd-aversion problem illustrate the proposed framework.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 61 references that are in the map. First posted on arXiv 25 Jun 2026.
Builds on
42 of 61 references.
References · 42
- 2019A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problemscited by 95
- 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
- 2023Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learningcited by 72
- 2022q-Learning in Continuous Timecited by 68
- 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
- 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 56
- 2019Making Deep Q-learning methods robust to time discretizationcited by 51
- 2020Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysiscited by 44
- 2020Unified Reinforcement Q-Learning for Mean Field Game and Control Problemscited by 42
- 2023Policy Optimization for Continuous Reinforcement Learningcited by 30
- 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 29
- 2019Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methodscited by 24
30 more
- 2019Mean-field Markov decision processes with common noise and open-loop controlscited by 24
- 2019Dynamic Programming Principles for Mean-Field Controls with Learningcited by 22
- 2021Numerical Methods for Mean Field Games and Mean Field Type Controlcited by 21
- 2023Actor-Critic learning for mean-field control in continuous timecited by 19
- 2023Continuous-time q-learning for mean-field control problemscited by 17
- 2022Mean-field neural networks: learning mappings on Wasserstein spacecited by 15
- 2016Dynamic programming for optimal control of stochastic McKean-Vlasov dynamicscited by 11
- 2016Limit theory for controlled McKean-Vlasov dynamicscited by 11
- 2023Fast Policy Learning for Linear Quadratic Control with Entropy Regularizationcited by 11
- 2021Mean Field Markov Decision Processescited by 9
- 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learningcited by 9
- 2023Actor critic learning algorithms for mean-field control with moment neural networkscited by 6
- 2016Mean Field Type Control with Congestion (II): An Augmented Lagrangian Methodcited by 5
- 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundationscited by 5
- 2024On Bellman equations for continuous-time policy evaluation I: discretization and approximationcited by 4
- 2024Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noisecited by 4
- 2025Learning algorithms for mean field optimal controlcited by 4
- 2026Policy Gradient for Continuous-Time Mean-Field Controlcited by 4
- 2017A numerical scheme for a mean field game in some queueing systems based on Markov chain approximation methodcited by 3
- 2024PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluationcited by 3
- 2025A Temporal Difference Method for Stochastic Continuous Dynamicscited by 3
- 2026Model-free policy gradient for discrete-time mean-field controlcited by 3
- 2016Randomized dynamic programming principle and Feynman-Kac representation for optimal control of McKean-Vlasov dynamicscited by 2
- 2023Comparison of viscosity solutions for a class of second order PDEs on the Wasserstein spacecited by 2
- 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Controlcited by 2
- 2026Discretization error from regularized Reinforcement Learning to continuous-time stochastic controlcited by 2
- 2016Rate Control under Heavy Traffic with Strategic Serverscited by 1
- 2024Learning with Linear Function Approximations in Mean-Field Controlcited by 1
- 2025Comparison for semi-continuous viscosity solutions for second order PDEs on the Wasserstein spacecited by 1
- 2025Optimal-PhiBE: A PDE-based Model-free framework for Continuous-time Reinforcement Learningcited by 1
Built on by
Not cited yet.