# Temporal Difference Learning with Continuous Time and State in the Stochastic Setting

arXiv: https://arxiv.org/abs/2202.07960 · HTML: https://mlsanity.com/paper/2202.07960/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 49 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 13 of 13)

- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)
- [Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)](https://mlsanity.com/paper/1306.2119/) (2013)
- [Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations](https://mlsanity.com/paper/1709.05963/) (2017)
- [An overview on deep learning-based approximation methods for partial differential equations](https://mlsanity.com/paper/2012.12348/) (2020)
- [Making Deep Q-learning methods robust to time discretization](https://mlsanity.com/paper/1901.09732/) (2019)
- [Deep Learning Approximation for Stochastic Control Problems](https://mlsanity.com/paper/1611.07422/) (2016)
- [Continuous-Time Model-Based Reinforcement Learning](https://mlsanity.com/paper/2102.04764/) (2021)
- [Neural networks-based algorithms for stochastic control and PDEs in finance](https://mlsanity.com/paper/2101.08068/) (2021)
- [Value Iteration in Continuous Actions, States and Time](https://mlsanity.com/paper/2105.04682/) (2021)
- [Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications](https://mlsanity.com/paper/1812.05916/) (2018)

## Built on by (most cited first, 5 of 5)

- [Recent Developments in Machine Learning Methods for Stochastic Control and Games](https://mlsanity.com/paper/2303.10257/) (2023)
- [On Bellman equations for continuous-time policy evaluation I: discretization and approximation](https://mlsanity.com/paper/2407.05966/) (2024)
- [Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control](https://mlsanity.com/paper/2504.21793/) (2025)
- [A Temporal Difference Method for Stochastic Continuous Dynamics](https://mlsanity.com/paper/2505.15544/) (2025)
- [Fast and Robust Convergence Rate for TD(0) with Linear Function Approximation, Universal Learning Steps and I.I.D. Samples](https://mlsanity.com/paper/2606.05967/) (2026)

## Abstract (arXiv)

We consider the problem of continuous-time policy evaluation. This consists in learning through observations the value function associated with an uncontrolled continuous-time stochastic dynamic and a reward function. We propose two original variants of the well-known TD(0) method using vanishing time steps. One is model-free and the other is model-based. For both methods, we prove theoretical convergence rates that we subsequently verify through numerical simulations. Alternatively, those methods can be interpreted as novel reinforcement learning approaches for approximating solutions of linear PDEs (partial differential equations) or linear BSDEs (backward stochastic differential equations).