Do Neural Networks Lose Plasticity in a Gradually Changing World?
Not on the map. Why?
Abstract (arXiv)
Continual learning has become a trending topic in machine learning. Recent studies have discovered an interesting phenomenon called loss of plasticity, referring to neural networks gradually losing the ability to learn new tasks. However, existing plasticity research largely relies on benchmarks with abrupt task transitions, without examining whether the abruptness itself contributes to the observed plasticity loss. In this paper, we investigate the role of transition abruptness by simulating gradually changing environments through input/output interpolation and task sampling. We perform theoretical and empirical analysis, showing that the severity of plasticity loss is closely tied to the abruptness of task transitions, and can be substantially reduced when the environment changes gradually.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 9 Feb 2026.
Builds on
33 of 56 references.
References · 33
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2017Continual Learning with Deep Generative Replaycited by 1,152
- 2019When Does Label Smoothing Help?cited by 985
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
- 2017Sharp Minima Can Generalize For Deep Netscited by 462
- 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
- 2021Are Transformers More Robust Than CNNs?cited by 187
- 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
- 2023Understanding plasticity in neural networkscited by 170
21 more
- 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
- 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 118
- 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
- 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 79
- 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 63
- 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
- 2021A study on the plasticity of neural networkscited by 44
- 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
- 2024Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networkscited by 40
- 2023Small batch deep reinforcement learningcited by 35
- 2024Make Continual Learning Stronger via C-Flatcited by 35
- 2024Learning Continually by Spectral Regularizationcited by 34
- 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
- 2023Resetting the Optimizer in Deep RL: An Empirical Studycited by 26
- 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
- 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
- 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
- 2024Plastic Learning with Deep Fourier Featurescited by 14
- 2024DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticitycited by 7
- 2023Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learningcited by 6
- 2025Activation Function Design Sustains Plasticity in Continual Learningcited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.