# The Impact of Reinitialization on Generalization in Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2109.00267 · HTML: https://mlsanity.com/paper/2109.00267/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [MNIST-C: A Robustness Benchmark for Computer Vision](https://mlsanity.com/paper/1906.02337/) (2019)
- [What Do Neural Networks Learn When Trained With Random Labels?](https://mlsanity.com/paper/2006.10455/) (2020)
- [RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr](https://mlsanity.com/paper/2007.03349/) (2020)
- [Knowledge Evolution in Neural Networks](https://mlsanity.com/paper/2103.05152/) (2021)
- [iCassava 2019 Fine-Grained Visual Categorization Challenge](https://mlsanity.com/paper/1908.02900/) (2019)
- [DNN or k-NN: That is the Generalize vs. Memorize Question](https://mlsanity.com/paper/1805.06822/) (2018)

## Built on by (most cited first, 20 of 20)

- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [The Dormant Neuron Phenomenon in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.12902/) (2023)
- [Improving Language Plasticity via Pretraining with Active Forgetting](https://mlsanity.com/paper/2307.01163/) (2023)
- [Diagnosing and Re-learning for Balanced Multimodal Learning](https://mlsanity.com/paper/2407.09705/) (2024)
- [Fortuitous Forgetting in Connectionist Networks](https://mlsanity.com/paper/2202.00155/) (2022)
- [Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting](https://mlsanity.com/paper/2406.00053/) (2024)
- [When Does Re-initialization Work?](https://mlsanity.com/paper/2206.10011/) (2022)
- [Learn, Unlearn and Relearn: An Online Learning Paradigm for Deep Neural Networks](https://mlsanity.com/paper/2303.10455/) (2023)
- [ReFine: Re-randomization before Fine-tuning for Cross-domain Few-shot Learning](https://mlsanity.com/paper/2205.05282/) (2022)
- [Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learning](https://mlsanity.com/paper/2310.07996/) (2023)
- [Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training](https://mlsanity.com/paper/2401.14948/) (2024)
- [Reinitializing weights vs units for maintaining plasticity in neural networks](https://mlsanity.com/paper/2508.00212/) (2025)
- [Robust Ante-hoc Graph Explainer using Bilevel Optimization](https://mlsanity.com/paper/2305.15745/) (2023)
- [Breaking the Reclustering Barrier in Centroid-based Deep Clustering](https://mlsanity.com/paper/2411.02275/) (2024)
- [Towards Cross Domain Generalization of Hamiltonian Representation via Meta Learning](https://mlsanity.com/paper/2212.01168/) (2022)
- [Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2407.17120/) (2024)
- [The Effectiveness of Random Forgetting for Robust Generalization](https://mlsanity.com/paper/2402.11733/) (2024)
- [Improving Autoencoder Training Performance for Hyperspectral Unmixing with Network Reinitialisation](https://mlsanity.com/paper/2109.13748/) (2021)
- [Recovering Plasticity of Neural Networks via Soft Weight Rescaling](https://mlsanity.com/paper/2507.04683/) (2025)
- [CRAFT: Contextual Re-Activation of Filters for face recognition Training](https://mlsanity.com/paper/2312.00072/) (2023)

## Abstract (arXiv)

Recent results suggest that reinitializing a subset of the parameters of a neural network during training can improve generalization, particularly for small training sets. We study the impact of different reinitialization methods in several convolutional architectures across 12 benchmark image classification datasets, analyzing their potential gains and highlighting limitations. We also introduce a new layerwise reinitialization algorithm that outperforms previous methods and suggest explanations of the observed improved generalization. First, we show that layerwise reinitialization increases the margin on the training examples without increasing the norm of the weights, hence leading to an improvement in margin-based generalization bounds for neural networks. Second, we demonstrate that it settles in flatter local minima of the loss surface. Third, it encourages learning general rules and discourages memorization by placing emphasis on the lower layers of the neural network. Our takeaway message is that the accuracy of convolutional neural networks can be improved for small datasets using bottom-up layerwise reinitialization, where the number of reinitialized layers may vary depending on the available compute budget.