# Parameter-Level Soft-Masking for Continual Learning

arXiv: https://arxiv.org/abs/2306.14775 · HTML: https://mlsanity.com/paper/2306.14775/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 21 of 21)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [PathNet: Evolution Channels Gradient Descent in Super Neural Networks](https://mlsanity.com/paper/1701.08734/) (2017)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Class-incremental Learning via Deep Model Consolidation](https://mlsanity.com/paper/1903.07864/) (2019)
- [Uncertainty-based Continual Learning with Adaptive Regularization](https://mlsanity.com/paper/1905.11614/) (2019)
- [Scalable and Order-robust Continual Learning with Additive Parameter Decomposition](https://mlsanity.com/paper/1902.09432/) (2019)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [SpaceNet: Make Free Space For Continual Learning](https://mlsanity.com/paper/2007.07617/) (2020)
- [BinPlay: A Binary Latent Autoencoder for Generative Replay Continual Learning](https://mlsanity.com/paper/2011.14960/) (2020)
- [TAG: Task-based Accumulated Gradients for Lifelong learning](https://mlsanity.com/paper/2105.05155/) (2021)

## Built on by (most cited first, 30 of 40)

- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model](https://mlsanity.com/paper/2403.08350/) (2024)
- [Theory on Mixture-of-Experts in Continual Learning](https://mlsanity.com/paper/2406.16437/) (2024)
- [Make Continual Learning Stronger via C-Flat](https://mlsanity.com/paper/2404.00986/) (2024)
- [Overcoming Generic Knowledge Loss with Selective Parameter Update](https://mlsanity.com/paper/2308.12462/) (2023)
- [LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation](https://mlsanity.com/paper/2504.07448/) (2025)
- [TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidation](https://mlsanity.com/paper/2408.09857/) (2024)
- [Revisiting Neural Networks for Continual Learning: An Architectural Perspective](https://mlsanity.com/paper/2404.14829/) (2024)
- [Recurrent Knowledge Identification and Fusion for Language Model Continual Learning](https://mlsanity.com/paper/2502.17510/) (2025)
- [Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning](https://mlsanity.com/paper/2402.09542/) (2024)
- [Vector Quantization Prompting for Continual Learning](https://mlsanity.com/paper/2410.20444/) (2024)
- [KIF: Knowledge Identification and Fusion for Language Model Continual Learning](https://mlsanity.com/paper/2408.05200/) (2024)
- [Continual Learning in Vision-Language Models via Aligned Model Merging](https://mlsanity.com/paper/2506.03189/) (2025)
- [Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks](https://mlsanity.com/paper/2310.09436/) (2023)
- [In-context Continual Learning Assisted by an External Continual Learner](https://mlsanity.com/paper/2412.15563/) (2024)
- [Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay](https://mlsanity.com/paper/2404.10662/) (2024)
- [Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces](https://mlsanity.com/paper/2410.15698/) (2024)
- [Learning without Isolation: Pathway Protection for Continual Learning](https://mlsanity.com/paper/2505.18568/) (2025)
- [Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stages](https://mlsanity.com/paper/2603.12658/) (2026)
- [Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration](https://mlsanity.com/paper/2403.11373/) (2024)
- [Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates](https://mlsanity.com/paper/2512.04844/) (2025)
- [Towards Redundancy-Free Sub-networks in Continual Learning](https://mlsanity.com/paper/2312.00840/) (2023)
- [A Contrastive Symmetric Forward-Forward Algorithm (SFFA) for Continual Learning Tasks](https://mlsanity.com/paper/2409.07387/) (2024)
- [DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognition](https://mlsanity.com/paper/2502.00618/) (2025)
- [Investigating the Impact of Weight Sharing Decisions on Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2311.09506/) (2023)
- [Learning System Dynamics without Forgetting](https://mlsanity.com/paper/2407.00717/) (2024)
- [Temporal-Difference Variational Continual Learning](https://mlsanity.com/paper/2410.07812/) (2024)
- [Achieving Deep Continual Learning via Evolution](https://mlsanity.com/paper/2502.06210/) (2025)
- [Meta-Continual Learning of Neural Fields](https://mlsanity.com/paper/2504.05806/) (2025)
- [C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning](https://mlsanity.com/paper/2508.18860/) (2025)

## Abstract (arXiv)

Existing research on task incremental learning in continual learning has primarily focused on preventing catastrophic forgetting (CF). Although several techniques have achieved learning with no CF, they attain it by letting each task monopolize a sub-network in a shared network, which seriously limits knowledge transfer (KT) and causes over-consumption of the network capacity, i.e., as more tasks are learned, the performance deteriorates. The goal of this paper is threefold: (1) overcoming CF, (2) encouraging KT, and (3) tackling the capacity problem. A novel technique (called SPG) is proposed that soft-masks (partially blocks) parameter updating in training based on the importance of each parameter to old tasks. Each task still uses the full network, i.e., no monopoly of any part of the network by any task, which enables maximum KT and reduction in capacity usage. To our knowledge, this is the first work that soft-masks a model at the parameter-level for continual learning. Extensive experiments demonstrate the effectiveness of SPG in achieving all three objectives. More notably, it attains significant transfer of knowledge not only among similar tasks (with shared knowledge) but also among dissimilar tasks (with little shared knowledge) while mitigating CF.