# Statistical guarantees for continuous-time policy evaluation: blessing of ellipticity and new tradeoffs

arXiv: https://arxiv.org/abs/2502.04297 · HTML: https://mlsanity.com/paper/2502.04297/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2025.

## Built on by (most cited first, 4 of 4)

- [Is RL fine-tuning harder than regression? A PDE learning approach for diffusion models](https://mlsanity.com/paper/2509.02528/) (2025)
- [A Zeroth-Order Deep Learning Method for Fully Nonlinear Parabolic Partial Differential Equations with Unknown Coefficients](https://mlsanity.com/paper/2606.24999/) (2026)
- [Continuous-time reinforcement learning: ellipticity enables model-free value function approximation](https://mlsanity.com/paper/2602.06930/) (2026)
- [Provably Efficient Reinforcement Learning in Continuous-Time Episodic MDPs with Poisson Decision Epochs](https://mlsanity.com/paper/2609.23127/) (2026)

## Abstract (arXiv)

We study the estimation of the value function for continuous-time Markov diffusion processes using a single, discretely observed ergodic trajectory. Our work provides non-asymptotic statistical guarantees for the least-squares temporal-difference (LSTD) method, with performance measured in the first-order Sobolev norm. Specifically, the estimator attains an O(1 / √T) convergence rate when using a trajectory of length T; notably, this rate is achieved as long as T scales nearly linearly with both the mixing time of the diffusion and the number of basis functions employed. A key insight of our approach is that the ellipticity inherent in the diffusion process ensures robust performance even as the effective horizon diverges to infinity. Moreover, we demonstrate that the Markovian component of the statistical error can be controlled by the approximation error, while the martingale component grows at a slower rate relative to the number of basis functions. By carefully balancing these two sources of error, our analysis reveals novel trade-offs between approximation and statistical errors.