# Continual Learning with Node-Importance based Adaptive Group Sparse Regularization

arXiv: https://arxiv.org/abs/2003.13726 · HTML: https://mlsanity.com/paper/2003.13726/

## Key facts

- Cited by 105 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2020.

## Built on by (most cited first, 30 of 105)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2404.00228/) (2024)
- [Continual Learning and Catastrophic Forgetting](https://mlsanity.com/paper/2403.05175/) (2024)
- [AFEC: Active Forgetting of Negative Transfer in Continual Learning](https://mlsanity.com/paper/2110.12187/) (2021)
- [Representation Compensation Networks for Continual Semantic Segmentation](https://mlsanity.com/paper/2203.05402/) (2022)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers](https://mlsanity.com/paper/2301.11578/) (2023)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping](https://mlsanity.com/paper/2102.11343/) (2021)
- [CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One](https://mlsanity.com/paper/2207.06543/) (2022)
- [Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence](https://mlsanity.com/paper/2308.14991/) (2023)
- [NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks](https://mlsanity.com/paper/2206.09117/) (2022)
- [Continuum: Simple Management of Complex Continual Learning Scenarios](https://mlsanity.com/paper/2102.06253/) (2021)
- [Audio-Visual Class-Incremental Learning](https://mlsanity.com/paper/2308.11073/) (2023)
- [Towards Efficient Replay in Federated Incremental Learning](https://mlsanity.com/paper/2403.05890/) (2024)
- [Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractor](https://mlsanity.com/paper/2204.13349/) (2022)
- [How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition](https://mlsanity.com/paper/2207.07730/) (2022)
- [Formalizing the Generalization-Forgetting Trade-off in Continual Learning](https://mlsanity.com/paper/2109.14035/) (2021)
- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysis](https://mlsanity.com/paper/2104.09396/) (2021)
- [Personalized Federated Domain-Incremental Learning based on Adaptive Knowledge Matching](https://mlsanity.com/paper/2407.05005/) (2024)
- [Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors](https://mlsanity.com/paper/2104.13614/) (2021)
- [Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations](https://mlsanity.com/paper/2203.06514/) (2022)
- [Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting](https://mlsanity.com/paper/2502.02797/) (2025)
- [Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence](https://mlsanity.com/paper/2412.13779/) (2024)
- [Provable and Efficient Continual Representation Learning](https://mlsanity.com/paper/2203.02026/) (2022)
- [MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging](https://mlsanity.com/paper/2505.11883/) (2025)
- [Provable Contrastive Continual Learning](https://mlsanity.com/paper/2405.18756/) (2024)
- [Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models](https://mlsanity.com/paper/2505.15424/) (2025)
- [t-DGR: A Trajectory-Based Deep Generative Replay Method for Continual Learning in Decision Making](https://mlsanity.com/paper/2401.02576/) (2024)

## Abstract (arXiv)

We propose a novel regularization-based continual learning method, dubbed as Adaptive Group Sparsity based Continual Learning (AGS-CL), using two group sparsity-based penalties. Our method selectively employs the two penalties when learning each node based its the importance, which is adaptively updated after learning each new task. By utilizing the proximal gradient descent method for learning, the exact sparsity and freezing of the model is guaranteed, and thus, the learner can explicitly control the model capacity as the learning continues. Furthermore, as a critical detail, we re-initialize the weights associated with unimportant nodes after learning each task in order to prevent the negative transfer that causes the catastrophic forgetting and facilitate efficient learning of new tasks. Throughout the extensive experimental results, we show that our AGS-CL uses much less additional memory space for storing the regularization parameters, and it significantly outperforms several state-of-the-art baselines on representative continual learning benchmarks for both supervised and reinforcement learning tasks.