mlsanityData to 7 Oct 2026

Interference and Generalization in Temporal Difference Learning

arXiv 2003.06350 ↗First posted 13 Mar 2020cs.LG stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

We study the link between generalization and interference in temporal-difference (TD) learning. Interference is defined as the inner product of two different gradients, representing their alignment. This quantity emerges as being of interest from a variety of observations about neural networks, parameter sharing and the dynamics of learning. We find that TD easily leads to low-interference, under-generalizing parameters, while the effect seems reversed in supervised learning. We hypothesize that the cause can be traced back to the interplay between the dynamics of interference and bootstrapping. This is supported empirically by several observations: the negative relationship between the generalization gap and interference in TD, the negative effect of bootstrapping on interference and the local coherence of targets, and the contrast between the propagation rate of information in TD(0) versus TD(λ) and regression tasks such as Monte-Carlo policy evaluation. We hope that these new findings can guide the future discovery of better bootstrapping methods.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 11; 2022: 7; 2023: 8; 2024: 4; 2025: 8; 2026: 2.

41 papers.

Sort:

By area:

  1. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  2. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  3. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  4. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  5. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  6. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  7. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 129
  8. 2022Generative Flow Networks for Discrete Probabilistic Modelingcited by 113
  9. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  10. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  11. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  12. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
29 more
  1. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  2. 2026Maximum Likelihood Reinforcement Learningcited by 29
  3. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  4. 2021Quantum Architecture Search via Continual Reinforcement Learningcited by 22
  5. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  6. 2022Adversarial Cheap Talkcited by 16
  7. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  8. 2021Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillationcited by 15
  9. 2021Correcting Momentum in Temporal Difference Learningcited by 12
  10. 2023Learning to Optimize for Reinforcement Learningcited by 12
  11. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  12. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
  13. 2023Continual Task Allocation in Meta-Policy Network via Sparse Promptingcited by 11
  14. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  15. 2021Disparity Between Batches as a Signal for Early Stoppingcited by 9
  16. 2024Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignmentcited by 9
  17. 2022Rethinking Value Function Learning for Generalization in Reinforcement Learningcited by 8
  18. 2021Preferential Temporal Difference Learningcited by 6
  19. 2023Online Reinforcement Learning in Non-Stationary Context-Driven Environmentscited by 6
  20. 2024Investigating Generalization Behaviours of Generative Flow Networkscited by 6
  21. 2023Measuring and Mitigating Interference in Reinforcement Learningcited by 5
  22. 2025Imagined Autocurriculacited by 4
  23. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learningcited by 3
  24. 2024Continual Task Learning through Adaptive Policy Self-Compositioncited by 1
  25. 2025Is Exploration or Optimization the Problem for Deep Reinforcement Learning?cited by 1
  26. 2023Minibatch training of neural network ensembles via trajectory samplingcited by 0
  27. 2025Teleology-Driven Affective Computing: A Causal Framework for Sustained Well-Beingcited by 0
  28. 2025Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Datacited by 0
  29. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0