mlsanityData to 7 Oct 2026

On the Performance of Temporal Difference Learning With Neural Networks

arXiv 2312.05397 ↗First posted 8 Dec 2023cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Neural Temporal Difference (TD) Learning is an approximate temporal difference method for policy evaluation that uses a neural network for function approximation. Analysis of Neural TD Learning has proven to be challenging. In this paper we provide a convergence analysis of Neural TD Learning with a projection onto B(θ0, ω), a ball of fixed radius ω around the initial point θ0. We show an approximation bound of O(ε) + tildeO (1/√m) where ε is the approximation quality of the best neural network in B(θ0, ω) and m is the width of all hidden layers in the network.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 12 of its 26 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

12 of 26 references.

References · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  5. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  6. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  7. 2018On Markov Chain Gradient Descentcited by 63
  8. 2019A Finite-Time Analysis of Q-Learning with Neural Network Function Approximationcited by 47
  9. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  10. 2020Temporal Difference Learning as Gradient Splittingcited by 21
  11. 2018Approximate Temporal Difference Learning is a Gradient Descent for Reversible Policiescited by 10
  12. 2021Sample Complexity and Overparameterization Bounds for Temporal Difference Learning with Neural Network Approximationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 2.

6 papers.

  1. 2024Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Samplingcited by 6
  2. 2024One-Shot Averaging for Distributed TD(λ) Under Markov Samplingcited by 4
  3. 2024An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networkscited by 3
  4. 2025A Communication-Efficient Decentralized Actor-Critic Algorithmcited by 1
  5. 2026Bridging the Gap Between Average and Discounted TD Learningcited by 0
  6. 2026Beyond the Independence Assumption: Finite-Sample Guarantees for Deep Q-Learning under τ-Mixingcited by 0