Optimal-PhiBE: A PDE-based Model-free framework for Continuous-time Reinforcement Learning
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
This paper addresses continuous-time reinforcement learning (CTRL) where the system dynamics are governed by an unknown stochastic differential equation, and only discrete-time observations are available. Existing approaches face limitations: model-based PDE methods suffer from non-identifiability, while model-free methods based on the discrete-time optimal Bellman equation (Optimal-BE) suffer from large discretization errors that are highly sensitive to both the system dynamics and the reward structure. To overcome these challenges, we introduce Optimal-PhiBE, a formulation that integrates discrete-time information into a continuous-time PDE, combining the strength of both existing frameworks while mitigating their limitations. Optimal-PhiBE exhibits smaller discretization errors when the uncontrolled system evolves slowly, and demonstrates reduced sensitivity to oscillatory reward structures, and enables model-free algorithms that bypass explicit dynamics estimation. In the linear-quadratic regulator (LQR) setting, sharp error bounds are established for both Optimal-PhiBE and Optimal-BE. The results show that Optimal-PhiBE exactly recovers the optimal policy in the undiscounted case and substantially outperforms Optimal-BE when the problem is weakly discounted or control-dominant. Furthermore, we extend Optimal-PhiBE to higher orders, providing increasingly accurate approximations. A model-free policy iteration algorithm is proposed to solve the Optimal-PhiBE directly from trajectory data. Numerical experiments are conducted to verify the theoretical findings.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. First posted on arXiv 5 Jun 2025.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Discretization error from regularized Reinforcement Learning to continuous-time stochastic controlcited by 2
- 2026A Zeroth-Order Deep Learning Method for Fully Nonlinear Parabolic Partial Differential Equations with Unknown Coefficientscited by 2
- 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Datacited by 1
- 2026PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Controlcited by 0
- 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data25 Jun 2026
- 2026A Zeroth-Order Deep Learning Method for Fully Nonlinear Parabolic Partial Differential Equations with Unknown Coefficients23 Jun 2026
- 2026PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control20 Jun 2026
- 2026Discretization error from regularized Reinforcement Learning to continuous-time stochastic control23 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2024PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluation
- 2025A Temporal Difference Method for Stochastic Continuous Dynamics
- 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
- 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation
- 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
- 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models
- 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms
- 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control