mlsanityData to 7 Oct 2026

Learning Dynamics and Generalization in Reinforcement Learning

arXiv 2206.02126 ↗First posted 5 Jun 2022cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Solving a reinforcement learning (RL) problem poses two competing challenges: fitting a potentially discontinuous value function, and generalizing well to new observations. In this paper, we analyze the learning dynamics of temporal difference algorithms to gain novel insight into the tension between these two objectives. We show theoretically that temporal difference learning encourages agents to fit non-smooth components of the value function early in training, and at the same time induces the second-order effect of discouraging generalization. We corroborate these findings in deep RL agents trained on a range of environments, finding that neural networks trained using temporal difference algorithms on dense reward tasks exhibit weaker generalization between states than randomly initialized networks and networks trained with policy gradient methods. Finally, we investigate how post-training policy distillation may avoid this pitfall, and show that this approach improves generalization to novel environments in the ProcGen suite and improves robustness to input perturbations.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 45 of its 66 references that are in the map. First posted on arXiv 5 Jun 2022.

Builds on

45 of 66 references.

References · 45

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  8. 2020Reinforcement Learning with Augmented Datacited by 452
  9. 2015Policy Distillationcited by 414
  10. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  11. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  12. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
33 more
  1. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  2. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  3. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  4. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  5. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  6. 2020Implicit Gradient Regularizationcited by 145
  7. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  8. 2020Phasic Policy Gradientcited by 121
  9. 2018Generalization and Regularization in DQNcited by 117
  10. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  11. 2020Invariant Causal Prediction for Block MDPscited by 104
  12. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  13. 2019Observational Overfitting in Reinforcement Learningcited by 90
  14. 2019Distilling Policy Distillationcited by 81
  15. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  16. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  17. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  18. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  19. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  20. 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 50
  21. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  22. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  23. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  24. 2020Interference and Generalization in Temporal Difference Learningcited by 41
  25. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
  26. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  27. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 28
  28. 2020GRAC: Self-Guided and Self-Regularized Actor-Criticcited by 20
  29. 2020Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networkscited by 17
  30. 2020On Catastrophic Interference in Atari 2600 Gamescited by 13
  31. 2019Generalizing from a few environments in safety-critical reinforcement learningcited by 8
  32. 2020Towards a practical measure of interference for reinforcement learningcited by 6
  33. 2019Overcoming Catastrophic Interference in Online Reinforcement Learning with Dynamic Self-Organizing Mapscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 3; 2025: 2; 2026: 1.

11 papers.

  1. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  2. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  3. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  4. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 13
  5. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  6. 2023Representations and Exploration for Deep Reinforcement Learning using Singular Value Decompositioncited by 2
  7. 2022RPM: Generalizable Behaviors for Multi-Agent Reinforcement Learningcited by 1
  8. 2025Universal Value-Function Uncertaintiescited by 1
  9. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0
  10. 2025Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Datacited by 0
  11. 2026On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inferencecited by 0