mlsanityData to 7 Oct 2026

Gauss-Newton Temporal Difference Learning with Nonlinear Function Approximation

arXiv 2302.13087 ↗First posted 25 Feb 2023math.OC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, a Gauss-Newton Temporal Difference (GNTD) learning method is proposed to solve the Q-learning problem with nonlinear function approximation. In each iteration, our method takes one Gauss-Newton (GN) step to optimize a variant of Mean-Squared Bellman Error (MSBE), where target networks are adopted to avoid double sampling. Inexact GN steps are analyzed so that one can safely and efficiently compute the GN updates by cheap matrix iterations. Under mild conditions, non-asymptotic finite-sample convergence to the globally optimal Q function is derived for various nonlinear function approximations. In particular, for neural network parameterization with relu activation, GNTD achieves an improved sample complexity of tildeO(varepsilon-1), as opposed to the mathcalO(varepsilon-2) sample complexity of the existing neural TD methods. An tildeO(varepsilon-1.5) sample complexity of GNTD is also established for general smooth function approximations. We validate our method via extensive experiments in several RL benchmarks, where GNTD exhibits both higher rewards and faster convergence than TD-type methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 25 Feb 2023.

Builds on

30 of 43 references.

References · 30

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  10. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  11. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  12. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
18 more
  1. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  2. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  3. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  4. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  5. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 165
  6. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  7. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  8. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  9. 2019A Finite-Time Analysis of Q-Learning with Neural Network Function Approximationcited by 47
  10. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  11. 2020Gradient Temporal-Difference Learning with Regularized Correctionscited by 38
  12. 2017Least-Squares Temporal Difference Learning for the Linear Quadratic Regulatorcited by 27
  13. 2019Neural Temporal-Difference and Q-Learning Provably Converge to Global Optimacited by 20
  14. 2020Sample Complexity Bounds for Two Timescale Value-based Reinforcement Learning Algorithmscited by 19
  15. 2019Geometric Insights into the Convergence of Nonlinear TD Learningcited by 12
  16. 2016Accelerated Gradient Temporal Difference Learningcited by 10
  17. 2021Non-Asymptotic Analysis for Two Time-scale TDC with General Smooth Function Approximationcited by 8
  18. 2019Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimescited by 4

Built on by

Not cited yet.