# Residual Continual Learning

arXiv: https://arxiv.org/abs/2002.06774 · HTML: https://mlsanity.com/paper/2002.06774/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Feb 2020.

## Built on by (most cited first, 11 of 11)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Continual Learning for Fake Audio Detection](https://mlsanity.com/paper/2104.07286/) (2021)
- [BECAME: BayEsian Continual Learning with Adaptive Model MErging](https://mlsanity.com/paper/2504.02666/) (2025)
- [Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications](https://mlsanity.com/paper/2410.09218/) (2024)
- [Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning](https://mlsanity.com/paper/2505.22389/) (2025)
- [Exemplar-Free Continual Transformer with Convolutions](https://mlsanity.com/paper/2308.11357/) (2023)
- [Transferability Estimation Based On Principal Gradient Expectation](https://mlsanity.com/paper/2211.16299/) (2022)
- [Chunking: Continual Learning is not just about Distribution Shift](https://mlsanity.com/paper/2310.02206/) (2023)
- [A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization](https://mlsanity.com/paper/2112.12299/) (2021)
- [Residual Multi-Task Learner for Applied Ranking](https://mlsanity.com/paper/2411.09705/) (2024)
- [GMM-IL: Image Classification using Incrementally Learnt, Independent Probabilistic Models for Small Sample Sizes](https://mlsanity.com/paper/2212.00572/) (2022)

## Abstract (arXiv)

We propose a novel continual learning method called Residual Continual Learning (ResCL). Our method can prevent the catastrophic forgetting phenomenon in sequential learning of multiple tasks, without any source task information except the original network. ResCL reparameterizes network parameters by linearly combining each layer of the original network and a fine-tuned network; therefore, the size of the network does not increase at all. To apply the proposed method to general convolutional neural networks, the effects of batch normalization layers are also considered. By utilizing residual-learning-like reparameterization and a special weight decay loss, the trade-off between source and target performance is effectively controlled. The proposed method exhibits state-of-the-art performance in various continual learning scenarios.