mlsanityData to 14 Oct 2026

A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise

arXiv 2606.18183 ↗First posted 16 Jun 2026stat.ML cs.LG math.PRcited by 0

Not on the map. Why?

Abstract (arXiv)

Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asymptotic mean dynamics but neglects stochastic fluctuations determining the error floor. We introduce a stochastic differential equation (SDE) approximation for linear TD(0) under Markovian noise. The resulting model distinguishes the contraction dynamics governed by the projected Bellman operator from the influence of Markovian sampling. As a consequence, the model explains the constant-stepsize error floor through the interaction between Markovian long-run covariance and the contraction geometry of the projected Bellman operator.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

22 of 32 references.

References · 22

  1. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  2. 2017Three Factors Influencing Minima in SGDcited by 273
  3. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  4. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 166
  5. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  6. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  7. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  8. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  9. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  10. 2020Least Squares Regression with Markovian Data: Fundamental Limits and Algorithmscited by 56
  11. 2021Optimal and instance-dependent guarantees for Markovian linear stochastic approximationcited by 33
  12. 2024Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noisecited by 23
10 more
  1. 2014On TD(0) with function approximation: Concentration bounds and a centered variant with exponential convergencecited by 22
  2. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  3. 2018Continuous-time Models for Stochastic Optimization Algorithmscited by 17
  4. 2024Rates of Convergence in the Central Limit Theorem for Markov Chains, with an Application to TD Learningcited by 17
  5. 2024A Simple Finite-Time Analysis of TD Learning with Linear Function Approximationcited by 16
  6. 2025Statistical inference for Linear Stochastic Approximation with Markovian Noisecited by 13
  7. 2024SDEs for Minimax Optimizationcited by 6
  8. 2022On the Theoretical Properties of Noise Correlation in Stochastic Optimizationcited by 5
  9. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 5
  10. 2026Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspectivecited by 2

Built on by

Not cited yet.