# Why Do Neural Networks Forget: A Study of Collapse in Continual Learning

arXiv: https://arxiv.org/abs/2603.04580 · HTML: https://mlsanity.com/paper/2603.04580/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [How Well Do Self-Supervised Models Transfer?](https://mlsanity.com/paper/2011.13377/) (2020)
- [Self-Supervised Models are Continual Learners](https://mlsanity.com/paper/2112.04215/) (2021)
- [Delving Deeper into Convolutional Networks for Learning Video Representations](https://mlsanity.com/paper/1511.06432/) (2015)
- [VideoLSTM Convolves, Attends and Flows for Action Recognition](https://mlsanity.com/paper/1607.01794/) (2016)
- [A comprehensive, application-oriented study of catastrophic forgetting in DNNs](https://mlsanity.com/paper/1905.08101/) (2019)
- [SPACE: Structured Compression and Sharing of Representational Space for Continual Learning](https://mlsanity.com/paper/2001.08650/) (2020)
- [Hierarchically Structured Task-Agnostic Continual Learning](https://mlsanity.com/paper/2211.07725/) (2022)
- [Teamwork Is Not Always Good: An Empirical Study of Classifier Drift in Class-incremental Information Extraction](https://mlsanity.com/paper/2305.16559/) (2023)

## Abstract (arXiv)

Catastrophic forgetting is a major problem in continual learning, and lots of approaches arise to reduce it. However, most of them are evaluated through task accuracy, which ignores the internal model structure. Recent research suggests that structural collapse leads to loss of plasticity, as evidenced by changes in effective rank (eRank). This indicates a link to forgetting, since the networks lose the ability to expand their feature space to learn new tasks, which forces the network to overwrite existing representations. Therefore, in this study, we investigate the correlation between forgetting and collapse through the measurement of both weight and activation eRank. To be more specific, we evaluated four architectures, including MLP, ConvGRU, ResNet-18, and Bi-ConvGRU, in the split MNIST and Split CIFAR-100 benchmarks. Those models are trained through the SGD, Learning-without-Forgetting (LwF), and Experience Replay (ER) strategies separately. The results demonstrate that forgetting and collapse are strongly related, and different continual learning strategies help models preserve both capacity and performance in different efficiency.