mlsanityData to 7 Oct 2026

Correcting Momentum in Temporal Difference Learning

arXiv 2106.03955 ↗First posted 7 Jun 2021cs.LG stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

A common optimization tool used in deep reinforcement learning is momentum, which consists in accumulating and discounting past gradients, reapplying them at each iteration. We argue that, unlike in supervised learning, momentum in Temporal Difference (TD) learning accumulates gradients that become doubly stale: not only does the gradient of the loss change due to parameter updates, the loss itself changes due to bootstrapping. We first show that this phenomenon exists, and then propose a first-order correction term to momentum. We show that this correction term improves sample efficiency in policy evaluation by correcting target value drift. An important insight of this work is that deep RL methods are not always best served by directly importing techniques from the supervised setting.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 26 of its 56 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

26 of 56 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  9. 2020Revisiting Fundamentals of Experience Replaycited by 171
  10. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  11. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  12. 2018Generalization and Regularization in DQNcited by 117
14 more
  1. 2019BackPACK: Packing more into backpropcited by 77
  2. 2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
  3. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  4. 2020Interference and Generalization in Temporal Difference Learningcited by 41
  5. 2019Finite-Time Performance Bounds and Adaptive Learning Rate Selection for Two Time-Scale Reinforcement Learningcited by 40
  6. 2016Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descentcited by 26
  7. 2018The Utility of Sparse Representations for Control in Reinforcement Learningcited by 23
  8. 2019Deep Residual Reinforcement Learningcited by 21
  9. 2018Measuring and Characterizing Generalization in Deep Reinforcement Learningcited by 20
  10. 2019Momentum in Reinforcement Learningcited by 19
  11. 2016Accelerated Gradient Temporal Difference Learningcited by 10
  12. 2016Neural Taylor Approximations: Convergence and Exploration in Rectifier Networkscited by 9
  13. 2020Adaptive Temporal Difference Learning with Linear Function Approximationcited by 9
  14. 2020TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 1; 2023: 2; 2024: 4; 2025: 2; 2026: 1.

12 papers.

  1. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  2. 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
  3. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  4. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  5. 2021Continual Learning In Environments With Polynomial Mixing Timescited by 13
  6. 2023Learning to Optimize for Reinforcement Learningcited by 12
  7. 2024Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timestepscited by 11
  8. 2021Memory Augmented Optimizers for Deep Learningcited by 5
  9. 2024Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learningcited by 4
  10. 2024Beyond the Boundaries of Proximal Policy Optimizationcited by 3
  11. 2025ANO : Faster is Better in Noisy Landscapecited by 0
  12. 2026SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learningcited by 0