mlsanityData to 7 Oct 2026

A study on the plasticity of neural networks

arXiv 2106.00042 ↗First posted 31 May 2021cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

One aim shared by multiple settings, such as continual learning or transfer learning, is to leverage previously acquired knowledge to converge faster on the current task. Usually this is done through fine-tuning, where an implicit assumption is that the network maintains its plasticity, meaning that the performance it can reach on any given task is not affected negatively by previously seen tasks. It has been observed recently that a pretrained model on data from the same distribution as the one it is fine-tuned on might not reach the same generalisation as a freshly initialised one. We build and extend this observation, providing a hypothesis for the mechanics behind it. We discuss the implication of losing plasticity for continual learning which heavily relies on optimising pretrained models.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

15 of 21 references.

References · 15

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  5. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  6. 2018Rethinking ImageNet Pre-trainingcited by 376
  7. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  8. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  9. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  10. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  11. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  12. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
3 more
  1. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  2. 2019Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergencecited by 54
  3. 2019On Warm-Starting Neural Network Trainingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 7; 2024: 12; 2025: 17; 2026: 6.

44 papers.

Sort:

By area:

  1. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  2. 2023Understanding plasticity in neural networkscited by 170
  3. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  4. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  5. 2023Continual Learning: Applications and the Road Forwardcited by 59
  6. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  7. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  8. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  9. 2024Learning Continually by Spectral Regularizationcited by 34
  10. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  11. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  12. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 17
32 more
  1. 2024Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Resetcited by 17
  2. 2023Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learningcited by 16
  3. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 12
  4. 2024DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticitycited by 7
  5. 2025What Can Grokking Teach Us About Learning Under Nonstationarity?cited by 7
  6. 2026FIRE: Frobenius-Isometry Reinitialization for Balancing the Stability-Plasticity Tradeoffcited by 7
  7. 2025Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Losscited by 6
  8. 2025Adapting to Change: A Comparison of Continual and Transfer Learning for Modeling Building Thermal Dynamics under Concept Driftscited by 6
  9. 2025Activation Function Design Sustains Plasticity in Continual Learningcited by 6
  10. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 6
  11. 2024Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neuronscited by 5
  12. 2025Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticitycited by 5
  13. 2024Breaking the Reclustering Barrier in Centroid-based Deep Clusteringcited by 4
  14. 2025Variance Control via Weight Rescaling in LLM Pre-trainingcited by 4
  15. 2026Weight Decay Improves Language Model Plasticitycited by 4
  16. 2024Reset & Distill: A Recipe for Overcoming Negative Transfer in Continual Reinforcement Learningcited by 3
  17. 2025Continual learning for rotating machinery fault diagnosis with cross-domain environmental and operational variationscited by 3
  18. 2024Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsalcited by 2
  19. 2025Sparsity-Driven Plasticity in Multi-Task Reinforcement Learningcited by 2
  20. 2025AltNet: Addressing the Plasticity-Stability Dilemma in Reinforcement Learningcited by 2
  21. 2026Do Neural Networks Lose Plasticity in a Gradually Changing World?cited by 2
  22. 2025Recovering Plasticity of Neural Networks via Soft Weight Rescalingcited by 1
  23. 2025A Unified Noise-Curvature View of Loss of Trainabilitycited by 1
  24. 2025Intrinsic preservation of plasticity in continual quantum learningcited by 1
  25. 2026Portable Semantics, Private Dialects: Reuse and Negative Transfer in Latent Communication Between Language-Model Cellscited by 1
  26. 2021Does Optimal Source Task Performance Imply Optimal Pre-training for a Target Task?cited by 0
  27. 2022Generalization Through the Lens of Learning Dynamicscited by 0
  28. 2025Staying Alive: Online Neural Network Maintenance and Systemic Driftcited by 0
  29. 2025A Simple Baseline for Stable and Plastic Neural Networkscited by 0
  30. 2025On the Occurence of Critical Learning Periods in Neural Networkscited by 0
  31. 2026Meta-learning accelerates detector design optimizationcited by 0
  32. 2026When does a network's training history predict its future learning better than its current state? Evidence from a response probe and a forecasting screencited by 0