# Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data arXiv: https://arxiv.org/abs/2606.26498 ยท HTML: https://mlsanity.com/paper/2606.26498/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. - Builds on 42 of its 61 references that are in the map. - First posted on arXiv 25 Jun 2026. ## Placed in - Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/) ## Builds on (most cited first, 30 of 42) - [Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning](https://mlsanity.com/paper/2310.10943/) (2023) - [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021) - [A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems](https://mlsanity.com/paper/1912.01825/) (2019) - [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022) - [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021) - [Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning](https://mlsanity.com/paper/1910.12802/) (2019) - [Making Deep Q-learning methods robust to time discretization](https://mlsanity.com/paper/1901.09732/) (2019) - [Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis](https://mlsanity.com/paper/2002.04131/) (2020) - [Unified Reinforcement Q-Learning for Mean Field Game and Control Problems](https://mlsanity.com/paper/2006.13912/) (2020) - [Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls](https://mlsanity.com/paper/2010.14087/) (2020) - [Policy Optimization for Continuous Reinforcement Learning](https://mlsanity.com/paper/2305.18901/) (2023) - [Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods](https://mlsanity.com/paper/1910.04295/) (2019) - [Mean-field Markov decision processes with common noise and open-loop controls](https://mlsanity.com/paper/1912.07883/) (2019) - [Dynamic Programming Principles for Mean-Field Controls with Learning](https://mlsanity.com/paper/1911.07314/) (2019) - [Numerical Methods for Mean Field Games and Mean Field Type Control](https://mlsanity.com/paper/2106.06231/) (2021) - [Actor-Critic learning for mean-field control in continuous time](https://mlsanity.com/paper/2303.06993/) (2023) - [Continuous-time q-learning for mean-field control problems](https://mlsanity.com/paper/2306.16208/) (2023) - [Mean-field neural networks: learning mappings on Wasserstein space](https://mlsanity.com/paper/2210.15179/) (2022) - [Fast Policy Learning for Linear Quadratic Control with Entropy Regularization](https://mlsanity.com/paper/2311.14168/) (2023) - [Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2503.09981/) (2025) - [Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics](https://mlsanity.com/paper/1604.04057/) (2016) - [Limit theory for controlled McKean-Vlasov dynamics](https://mlsanity.com/paper/1609.08064/) (2016) - [Mean Field Markov Decision Processes](https://mlsanity.com/paper/2106.08755/) (2021) - [Actor critic learning algorithms for mean-field control with moment neural networks](https://mlsanity.com/paper/2309.04317/) (2023) - [On Bellman equations for continuous-time policy evaluation I: discretization and approximation](https://mlsanity.com/paper/2407.05966/) (2024) - [Mean Field Type Control with Congestion (II): An Augmented Lagrangian Method](https://mlsanity.com/paper/1611.02023/) (2016) - [PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluation](https://mlsanity.com/paper/2405.12535/) (2024) - [Full error analysis of policy gradient learning algorithms for exploratory linear quadratic mean-field control problem in continuous time with common noise](https://mlsanity.com/paper/2408.02489/) (2024) - [Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations](https://mlsanity.com/paper/2604.27372/) (2026) - [Learning algorithms for mean field optimal control](https://mlsanity.com/paper/2503.17869/) (2025) ## Built on by (most cited first, 1 of 1) - [Mean Field Reinforcement Learning](https://mlsanity.com/paper/2607.01525/) (2026) ## Abstract (arXiv) This paper develops a model-free framework for continuous-time mean-field control when the population evolves according to unknown controlled McKean--Vlasov dynamics and only discrete-time transition data are available. Model-based mean-field control requires the continuous-time drift and diffusion coefficients, which are not directly observed from fixed-step transitions, while a direct reduction to a discrete-time Bellman equation loses the continuous-time generator structure. To bridge these two viewpoints, we introduce a Mean-Field-PhiBE (MF-PhiBE), which incorporates discrete-time transition information into a continuous-time PDE on the Wasserstein space. The MF-PhiBE replaces the unknown infinitesimal drift and covariance in the policy-evaluation equation by one-step estimators computed from data, while preserving the generator structure of the McKean-Vlasov HJB equation. We also derive a policy-gradient theorem for entropy-regularized randomized feedback policies, expressing the actor direction through an action-wise infinitesimal advantage and the score of the policy. Combining these two ingredients yields a model-free actor-critic method. We prove a first-order consistency estimate showing that the value induced by an optimal MF-PhiBE policy approximates the optimal continuous-time value as the observation time step vanishes. For entropy-regularized LQR, we establish first-order policy convergence and second-order value convergence; under suitable conditions, the population-averaged feedback means coincide exactly. Numerical experiments on an LQR benchmark and a crowd-aversion problem illustrate the proposed framework.