mlsanityData to 7 Oct 2026

On Bellman equations for continuous-time policy evaluation I: discretization and approximation

arXiv 2407.05966 ↗First posted 8 Jul 2024cs.LG cs.NA math.NAcited by 4

Not on the map. Why?

Abstract (arXiv)

We study the problem of computing the value function from a discretely-observed trajectory of a continuous-time diffusion process. We develop a new class of algorithms based on easily implementable numerical schemes that are compatible with discrete-time reinforcement learning (RL) with function approximation. We establish high-order numerical accuracy as well as the approximation error guarantees for the proposed approach. In contrast to discrete-time RL problems where the approximation factor depends on the effective horizon, we obtain a bounded approximation factor using the underlying elliptic structures, even if the effective horizon diverges to infinity.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 16 of its 48 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

16 of 48 references.

References · 16

  1. 2021Physics-informed neural networks (PINNs) for fluid mechanics: A reviewcited by 470
  2. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  3. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  4. 2019A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problemscited by 95
  5. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  6. 2022q-Learning in Continuous Timecited by 68
  7. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
  8. 2020Is Temporal Difference Learning Optimal? An Instance-Dependent Analysiscited by 41
  9. 2021Solving PDE-constrained Control Problems Using Operator Learningcited by 41
  10. 2021Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimalitycited by 40
  11. 2021Optimal and instance-dependent guarantees for Markovian linear stochastic approximationcited by 33
  12. 2021Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networkscited by 27
4 more
  1. 2023Neural Operators for Bypassing Gain and Control Computations in PDE Backsteppingcited by 27
  2. 2022Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descentcited by 11
  3. 2022A Neural Network Approach for Stochastic Optimal Controlcited by 11
  4. 2022Temporal Difference Learning with Continuous Time and State in the Stochastic Settingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 18
  2. 2025Is RL fine-tuning harder than regression? A PDE learning approach for diffusion modelscited by 3
  3. 2026Continuous-time reinforcement learning: ellipticity enables model-free value function approximationcited by 0
  4. 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Datacited by 0