# The impact of model size on catastrophic forgetting in Online Continual Learning

arXiv: https://arxiv.org/abs/2407.00176 · HTML: https://mlsanity.com/paper/2407.00176/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [New Insights on Reducing Abrupt Representation Change in Online Continual Learning](https://mlsanity.com/paper/2104.05025/) (2021)
- [Avalanche: an End-to-End Library for Continual Learning](https://mlsanity.com/paper/2104.00405/) (2021)
- [Representational Continuity for Unsupervised Continual Learning](https://mlsanity.com/paper/2110.06976/) (2021)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [Benchmark Analysis of Representative Deep Neural Network Architectures](https://mlsanity.com/paper/1810.00736/) (2018)
- [Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data](https://mlsanity.com/paper/2108.09020/) (2021)
- [Model Complexity of Deep Learning: A Survey](https://mlsanity.com/paper/2103.05127/) (2021)
- [Comparing Different Deep Learning Architectures for Classification of Chest Radiographs](https://mlsanity.com/paper/2002.08991/) (2020)
- [A Comprehensive Empirical Evaluation on Online Continual Learning](https://mlsanity.com/paper/2308.10328/) (2023)
- [On the importance of cross-task features for class-incremental learning](https://mlsanity.com/paper/2106.11930/) (2021)
- [SCALE: Online Self-Supervised Lifelong Learning without Prior Knowledge](https://mlsanity.com/paper/2208.11266/) (2022)

## Built on by (most cited first, 2 of 2)

- [STAER: Temporal Aligned Rehearsal for Continual Spiking Neural Network](https://mlsanity.com/paper/2601.20870/) (2026)
- [Formalizing Forgetting: Theory of Forgetting for Online Continual Learning](https://mlsanity.com/paper/2610.14787/) (2026)

## Abstract (arXiv)

This study investigates the impact of model size on Online Continual Learning performance, with a focus on catastrophic forgetting. Employing ResNet architectures of varying sizes, the research examines how network depth and width affect model performance in class-incremental learning using the SplitCIFAR-10 dataset. Key findings reveal that larger models do not guarantee better Continual Learning performance; in fact, they often struggle more in adapting to new tasks, particularly in online settings. These results challenge the notion that larger models inherently mitigate catastrophic forgetting, highlighting the nuanced relationship between model size and Continual Learning efficacy. This study contributes to a deeper understanding of model scalability and its practical implications in Continual Learning scenarios.