mlsanityData to 7 Oct 2026

Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning

arXiv 2006.05826 ↗First posted 10 Jun 2020cs.LG cs.AI stat.MLcited by 75

Not on the map. Why?

Abstract (arXiv)

Non-stationarity can arise in Reinforcement Learning (RL) even in stationary environments. For example, most RL algorithms collect new data throughout training, using a non-stationary behaviour policy. Due to the transience of this non-stationarity, it is often not explicitly addressed in deep RL and a single neural network is continually updated. However, we find evidence that neural networks exhibit a memory effect where these transient non-stationarities can permanently impact the latent representation and adversely affect generalisation performance. Consequently, to improve generalisation of deep RL agents, we propose Iterated Relearning (ITER). ITER augments standard RL training by repeated knowledge transfer of the current policy into a freshly initialised network, which thereby experiences less non-stationarity during training. Experimentally, we show that ITER improves performance on the challenging generalisation benchmarks ProcGen and Multiroom.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 14; 2023: 19; 2024: 21; 2025: 12; 2026: 6.

75 papers.

Sort:

By area:

  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  2. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  3. 2023Understanding plasticity in neural networkscited by 170
  4. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  5. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  6. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  7. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  8. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  9. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 63
  10. 2021A study on the plasticity of neural networkscited by 44
  11. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  12. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
38 more
  1. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  2. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  3. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  4. 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 36
  5. 2024Learning Continually by Spectral Regularizationcited by 34
  6. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  7. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  8. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  9. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
  10. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  11. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  12. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  13. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  14. 2022Bayesian Generational Population-Based Trainingcited by 20
  15. 2022On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learningcited by 18
  16. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  17. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 17
  18. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  19. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  20. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  21. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  22. 2022When Does Re-initialization Work?cited by 12
  23. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  24. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 11
  25. 2022Learn the Time to Learn: Replay Scheduling in Continual Learningcited by 11
  26. 2024Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Controlcited by 11
  27. 2024Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timestepscited by 11
  28. 2022All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RLcited by 9
  29. 2023On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learningcited by 8
  30. 2024DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticitycited by 7
  31. 2025What Can Grokking Teach Us About Learning Under Nonstationarity?cited by 7
  32. 2022SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Explorationcited by 6
  33. 2022Forgetting and Imbalance in Robot Lifelong Learning with Off-policy Datacited by 5
  34. 2023Reinforcement Learning by Guided Safe Explorationcited by 5
  35. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
  36. 2025Spectral Collapse Drives Loss of Plasticity in Deep Continual Learningcited by 5
  37. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  38. 2022GriddlyJS: A Web IDE for Reinforcement Learningcited by 4

50 most cited of 75 All 75 (JSON)