# Restoring Neural Network Plasticity for Faster Transfer Learning

arXiv: https://arxiv.org/abs/2603.20860 · HTML: https://mlsanity.com/paper/2603.20860/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 21 Mar 2026.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)
- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [Understanding and Preventing Capacity Loss in Reinforcement Learning](https://mlsanity.com/paper/2204.09560/) (2022)
- [Self-Supervised Pretraining Improves Self-Supervised Pretraining](https://mlsanity.com/paper/2103.12718/) (2021)
- [Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNets](https://mlsanity.com/paper/2208.07463/) (2022)
- [K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning](https://mlsanity.com/paper/1810.10703/) (2018)
- [Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures](https://mlsanity.com/paper/1608.06037/) (2016)
- [Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2411.01713/) (2024)
- [Reinitializing weights vs units for maintaining plasticity in neural networks](https://mlsanity.com/paper/2508.00212/) (2025)
- [3D medical image segmentation with labeled and unlabeled data using autoencoders at the example of liver segmentation in CT images](https://mlsanity.com/paper/2003.07923/) (2020)

## Abstract (arXiv)

Transfer learning with models pretrained on ImageNet has become a standard practice in computer vision. Transfer learning refers to fine-tuning pretrained weights of a neural network on a downstream task, typically unrelated to ImageNet. However, pretrained weights can become saturated and may yield insignificant gradients, failing to adapt to the downstream task. This hinders the ability of the model to train effectively, and is commonly referred to as loss of neural plasticity. Loss of plasticity may prevent the model from fully adapting to the target domain, especially when the downstream dataset is atypical in nature. While this issue has been widely explored in continual learning, it remains relatively understudied in the context of transfer learning. In this work, we propose the use of a targeted weight re-initialization strategy to restore neural plasticity prior to fine-tuning. Our experiments show that both convolutional neural networks (CNNs) and vision transformers (ViTs) benefit from this approach, yielding higher test accuracy with faster convergence on several image classification benchmarks. Our method introduces negligible computational overhead and is compatible with common transfer learning pipelines.