# Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learning

arXiv: https://arxiv.org/abs/2310.07996 · HTML: https://mlsanity.com/paper/2310.07996/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [Meta-Learning Representations for Continual Learning](https://mlsanity.com/paper/1905.12588/) (2019)
- [Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning](https://mlsanity.com/paper/2003.04390/) (2020)
- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [Are All Layers Created Equal?](https://mlsanity.com/paper/1902.01996/) (2019)
- [Learning to Continually Learn](https://mlsanity.com/paper/2002.09571/) (2020)
- [Continual Backprop: Stochastic Gradient Descent with Persistent Randomness](https://mlsanity.com/paper/2108.06325/) (2021)
- [Fix your classifier: the marginal value of training the last weight layer](https://mlsanity.com/paper/1801.04540/) (2018)
- [Improving Language Plasticity via Pretraining with Active Forgetting](https://mlsanity.com/paper/2307.01163/) (2023)
- [Fortuitous Forgetting in Connectionist Networks](https://mlsanity.com/paper/2202.00155/) (2022)
- [RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr](https://mlsanity.com/paper/2007.03349/) (2020)
- [Dropout as an Implicit Gating Mechanism For Continual Learning](https://mlsanity.com/paper/2004.11545/) (2020)
- [The Impact of Reinitialization on Generalization in Convolutional Neural Networks](https://mlsanity.com/paper/2109.00267/) (2021)
- [Knowledge Evolution in Neural Networks](https://mlsanity.com/paper/2103.05152/) (2021)
- [When Does Re-initialization Work?](https://mlsanity.com/paper/2206.10011/) (2022)
- [Meta-Learning by the Baldwin Effect](https://mlsanity.com/paper/1806.07917/) (2018)
- [A Practitioners' Guide to Transfer Learning for Text Classification using Convolutional Neural Networks](https://mlsanity.com/paper/1801.06480/) (2018)
- [Learn, Unlearn and Relearn: An Online Learning Paradigm for Deep Neural Networks](https://mlsanity.com/paper/2303.10455/) (2023)

## Built on by (most cited first, 6 of 6)

- [Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn](https://mlsanity.com/paper/2506.00592/) (2025)
- [Learning, Fast and Slow: Towards LLMs That Adapt Continually](https://mlsanity.com/paper/2605.12484/) (2026)
- [Do Neural Networks Lose Plasticity in a Gradually Changing World?](https://mlsanity.com/paper/2602.09234/) (2026)
- [The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning](https://mlsanity.com/paper/2604.01913/) (2026)
- [How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks](https://mlsanity.com/paper/2507.01559/) (2025)
- [Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning](https://mlsanity.com/paper/2604.15414/) (2026)

## Abstract (arXiv)

This work identifies a simple pre-training mechanism that leads to representations exhibiting better continual and transfer learning. This mechanism -- the repeated resetting of weights in the last layer, which we nickname "zapping" -- was originally designed for a meta-continual-learning procedure, yet we show it is surprisingly applicable in many settings beyond both meta-learning and continual learning. In our experiments, we wish to transfer a pre-trained image classifier to a new set of classes, in a few shots. We show that our zapping procedure results in improved transfer accuracy and/or more rapid adaptation in both standard fine-tuning and continual learning settings, while being simple to implement and computationally efficient. In many cases, we achieve performance on par with state of the art meta-learning without needing the expensive higher-order gradients, by using a combination of zapping and sequential learning. An intuitive explanation for the effectiveness of this zapping procedure is that representations trained with repeated zapping learn features that are capable of rapidly adapting to newly initialized classifiers. Such an approach may be considered a computationally cheaper type of, or alternative to, meta-learning rapidly adaptable features with higher-order gradients. This adds to recent work on the usefulness of resetting neural network parameters during training, and invites further investigation of this mechanism.