mlsanityData to 14 Oct 2026

Reinitializing weights vs units for maintaining plasticity in neural networks

arXiv 2508.00212 ↗First posted 31 Jul 2025cs.NE cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Loss of plasticity is a phenomenon in which a neural network loses its ability to learn when trained for an extended time on non-stationary data. It is a crucial problem to overcome when designing systems that learn continually. An effective technique for preventing loss of plasticity is reinitializing parts of the network. In this paper, we compare two different reinitialization schemes: reinitializing units vs reinitializing weights. We propose a new algorithm, which we name textitselective weight reinitialization, for reinitializing the least useful weights in a network. We compare our algorithm to continual backpropagation and ReDo, two previously proposed algorithms that reinitialize units in the network. Through our experiments in continual supervised learning problems, we identify two settings when reinitializing weights is more effective at maintaining plasticity than reinitializing units: (1) when the network has a small number of units and (2) when the network includes layer normalization. Conversely, reinitializing weights and units are equally effective at maintaining plasticity when the network is of sufficient size and does not include layer normalization. We found that reinitializing weights maintains plasticity in a wider variety of settings than reinitializing units.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

37 of 56 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2016Layer Normalizationcited by 5,669
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  9. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  10. 2020What is the State of Neural Network Pruning?cited by 583
  11. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  12. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
25 more
  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  2. 2023DepGraph: Towards Any Structural Pruningcited by 203
  3. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  4. 2023Understanding plasticity in neural networkscited by 174
  5. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  6. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  7. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121
  8. 2021Continual Backprop: Stochastic Gradient Descent with Persistent Randomnesscited by 80
  9. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  10. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
  11. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  12. 2025Overtrained Language Models Are Harder to Fine-Tunecited by 58
  13. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 50
  14. 2023Maintaining Plasticity in Deep Continual Learningcited by 43
  15. 2024Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networkscited by 43
  16. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  17. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  18. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  19. 2021The Impact of Reinitialization on Generalization in Convolutional Neural Networkscited by 20
  20. 2021Knowledge Evolution in Neural Networkscited by 18
  21. 2023Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learningcited by 13
  22. 2023GVFs in the Real World: Making Predictions Online for Water Treatmentcited by 13
  23. 2022When Does Re-initialization Work?cited by 12
  24. 2023A Better Match for Drivers and Riders: Reinforcement Learning at Lyftcited by 9
  25. 2025Same accuracy, twice as fast: continuous training surpasses retraining from scratchcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Learning to Forget: Continual Learning with Adaptive Weight Decaycited by 1
  2. 2026Can Scale Save Us From Plasticity Loss in Large Language Models?cited by 1
  3. 2026Restoring Neural Network Plasticity for Faster Transfer Learningcited by 0
  4. 2026Attribution-Based Neuron Utility for Plasticity Restoration in Deep Networkscited by 0
  5. 2026Agentic Safety is an Epistemic Property, Not a Behavioral Onecited by 0
  6. 2026Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learningcited by 0