mlsanityData to 7 Oct 2026

Toward Efficient Gradient-Based Value Estimation

arXiv 2301.13757 ↗First posted 31 Jan 2023cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Gradient-based methods for value estimation in reinforcement learning have favorable stability properties, but they are typically much slower than Temporal Difference (TD) learning methods. We study the root causes of this slowness and show that Mean Square Bellman Error (MSBE) is an ill-conditioned loss function in the sense that its Hessian has large condition-number. To resolve the adverse effect of poor conditioning of MSBE on gradient based methods, we propose a low complexity batch-free proximal method that approximately follows the Gauss-Newton direction and is asymptotically robust to parameterization. Our main algorithm, called RANS, is efficient in the sense that it is significantly faster than the residual gradient methods while having almost the same computational complexity, and is competitive with TD on the classic problems that we tested.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 21 of its 65 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

21 of 65 references.

References · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2014New insights and perspectives on the natural gradient methodcited by 435
  6. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  7. 2017Stochastic Variance Reduction Methods for Policy Evaluationcited by 56
  8. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  9. 2020Gradient Temporal-Difference Learning with Regularized Correctionscited by 38
  10. 2020Finite-Sample Analysis of Proximal Gradient TD Algorithmscited by 32
  11. 2019Deep Residual Reinforcement Learningcited by 21
  12. 2014Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spacescited by 19
9 more
  1. 2013Distributed Policy Evaluation Under Multiple Behavior Strategiescited by 17
  2. 2019Geometric Insights into the Convergence of Nonlinear TD Learningcited by 12
  3. 2014Kalman Temporal Differencescited by 10
  4. 2021Convergent and Efficient Deep Q Network Algorithmcited by 8
  5. 2018Natural Gradient Deep Q-learningcited by 5
  6. 2021An Empirical Comparison of Off-policy Prediction Learning Algorithms in the Four Rooms Environmentcited by 3
  7. 2021Analysis and Optimisation of Bellman Residual Errors with Neural Function Approximationcited by 2
  8. 2021Gradient Temporal Difference with Momentum: Stability and Convergencecited by 2
  9. 2022Gradient Descent Temporal Difference-difference Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 0; 2026: 1.

4 papers.

  1. 2023A new Gradient TD Algorithm with only One Step-size: Convergence Rate Analysis using L-λ Smoothnesscited by 2
  2. 2026Optimistic Training and Convergence of Q-Learning -- Extended Versioncited by 1
  3. 2023Baird Counterexample is Solved: with an example of How to Debug a Two-time-scale Algorithmcited by 0
  4. 2024Fast and Accurate Neural Rendering Using Semi-Gradientscited by 0