mlsanityData to 7 Oct 2026

Gradient Iterated Temporal-Difference Learning

arXiv 2603.07833 ↗First posted 8 Mar 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent's long-term outcomes. Most approaches in this paradigm implement a semi-gradient update to boost the learning speed, which consists of ignoring the gradient of the bootstrapped estimate. While popular, this type of update is prone to divergence, as Baird's counterexample illustrates. Gradient TD methods were introduced to overcome this issue, but have not been widely used, potentially due to issues with learning speed compared to semi-gradient methods. Recently, iterated TD learning was developed to increase the learning speed of TD methods. For that, it learns a sequence of action-value functions in parallel, where each function is optimized to represent the application of the Bellman operator over the previous function in the sequence. While promising, this algorithm can be unstable due to its semi-gradient nature, as each function tracks a moving target. In this work, we modify iterated TD learning by computing the gradients over those moving targets, aiming to build a powerful gradient TD method that competes with semi-gradient methods. Our evaluation reveals that this algorithm, called Gradient Iterated Temporal-Difference learning, has a competitive learning speed against semi-gradient methods across various benchmarks, including Atari games, a result that no prior work on gradient TD methods has demonstrated.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 54 references that are in the map. First posted on arXiv 8 Mar 2026.

Builds on

31 of 54 references.

References · 31

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2015Prioritized Experience Replaycited by 1,070
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  9. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  10. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  11. 2020Munchausen Reinforcement Learningcited by 68
  12. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
19 more
  1. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  2. 2020Gradient Temporal-Difference Learning with Regularized Correctionscited by 38
  3. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  4. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  5. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  6. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  7. 2023TD Convergence: An Optimization Perspectivecited by 15
  8. 2025Deep Reinforcement Learning Agents are not even close to Human Intelligencecited by 8
  9. 2022Robust Losses for Learning Value Functionscited by 7
  10. 2018Nonlinear Distributional Gradient Temporal-Difference Learningcited by 6
  11. 2024Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learningcited by 6
  12. 2025Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learningcited by 6
  13. 2025Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learningcited by 6
  14. 2025Deep Reinforcement Learning with Gradient Eligibility Tracescited by 6
  15. 2021Learning Expected Emphatic Traces for Deep RLcited by 5
  16. 2025Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learningcited by 4
  17. 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
  18. 2022Chaining Value Functions for Off-Policy Learningcited by 3
  19. 2023Parameterized Projected Bellman Operatorcited by 3

Built on by

Not cited yet.