mlsanityData to 14 Oct 2026

Statistical Inference for Policy Evaluation with Temporal Difference Learning

arXiv 2410.16106 ↗First posted 21 Oct 2024stat.ML cs.LGcited by 15

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We investigate the statistical properties of Temporal Difference (TD) learning with Polyak-Ruppert averaging, arguably one of the most widely used algorithms in reinforcement learning, for the task of estimating the parameters of the optimal linear approximation to the value function. Assuming independent samples, we make three theoretical contributions that improve upon the current state-of-the-art results: (i) we establish refined high-dimensional Berry-Esseen bounds over the class of convex sets, achieving faster rates than the best known results, and (ii) we propose and analyze a novel, computationally efficient online plug-in estimator of the asymptotic covariance matrix; (iii) we derive sharper high probability convergence guarantees that depend explicitly on the asymptotic variance and hold under weaker conditions than those adopted in the literature. These results enable the construction of confidence regions and simultaneous confidence intervals for the linear parameters of the value function approximation, with guaranteed finite-sample coverage. We demonstrate the applicability of our theoretical findings through numerical experiments.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. First posted on arXiv 21 Oct 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descentcited by 10
  2. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 6
  3. 2025Improved Central Limit Theorem and Bootstrap Approximations for Linear Stochastic Approximationcited by 4
  4. 2025Statistical and Algorithmic Foundations of Reinforcement Learningcited by 2
  5. 2025On the Rate of Gaussian Approximation for Linear Regression Problemscited by 2
  6. 2026Optimism Stabilizes Thompson Sampling for Adaptive Inferencecited by 2
  7. 2025Statistical Inference for Linear Functionals of Online Least-squares SGD when t gtrsim d1+δcited by 1
  8. 2025Accelerated Distributional Temporal Difference Learning with Linear Function Approximationcited by 1
  9. 2026Gaussian Approximation for Asynchronous Q-learningcited by 1
  10. 2026Almost Sure Convergence Rates of Stochastic Approximation and Reinforcement Learning via a Poisson-Moreau Driftcited by 1
  11. 2026Self-Normalized Inference for Constant-Stepsize Temporal-Difference Learning under Markovian Samplingcited by 1
  12. 2026Online Inference in Distributional Temporal-Difference Learningcited by 1
3 more
  1. 2026Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalizationcited by 0
  2. 2026On Gaussian approximation for entropy-regularized Q-learning with function approximationcited by 0
  3. 2026Gaussian Approximation and Multiplier Bootstrap for Federated Linear Stochastic Approximationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Parameter-Free Temporal Difference Learning
  2. 2025Uncertainty quantification for Markov chain induced martingales with application to temporal difference learning
  3. 2024The surprising efficiency of temporal difference learning for rare event prediction
  4. 2024An MRP Formulation for Supervised Learning: Generalized Temporal Difference Learning Models
  5. 2026Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes
  6. 2025Adversarially-Robust TD Learning with Markovian Data: Finite-Time Rates and Fundamental Limits
  7. 2025Stabilizing Temporal Difference Learning via Implicit Stochastic Recursion
  8. 2024Statistical Efficiency of Distributional Temporal Difference Learning and Freedman's Inequality in Hilbert Spaces