# On the Soft-Subnetwork for Few-shot Class Incremental Learning

arXiv: https://arxiv.org/abs/2209.07529 · HTML: https://mlsanity.com/paper/2209.07529/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 69 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2004.10956/) (2020)
- [What's Hidden in a Randomly Weighted Neural Network?](https://mlsanity.com/paper/1911.13299/) (2019)
- [Supermasks in Superposition](https://mlsanity.com/paper/2006.14769/) (2020)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask](https://mlsanity.com/paper/1905.01067/) (2019)
- [Few-Shot Incremental Learning with Continually Evolved Classifiers](https://mlsanity.com/paper/2104.03047/) (2021)
- [Reinforced Continual Learning](https://mlsanity.com/paper/1805.12369/) (2018)
- [FearNet: Brain-Inspired Model for Incremental Learning](https://mlsanity.com/paper/1711.10563/) (2017)
- [Adaptive Aggregation Networks for Class-Incremental Learning](https://mlsanity.com/paper/2010.05063/) (2020)
- [Scalable and Order-robust Continual Learning with Additive Parameter Decomposition](https://mlsanity.com/paper/1902.09432/) (2019)
- [Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima](https://mlsanity.com/paper/2111.01549/) (2021)
- [Online Coreset Selection for Rehearsal-based Continual Learning](https://mlsanity.com/paper/2106.01085/) (2021)

## Built on by (most cited first, 30 of 37)

- [A Survey on Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2304.08130/) (2023)
- [Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models](https://mlsanity.com/paper/2311.08106/) (2023)
- [Compositional Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2405.17022/) (2024)
- [Rethinking Few-shot Class-incremental Learning: Learning from Yourself](https://mlsanity.com/paper/2407.07468/) (2024)
- [Non-Exemplar Online Class-incremental Continual Learning via Dual-prototype Self-augment and Refinement](https://mlsanity.com/paper/2303.10891/) (2023)
- [Calibrating Higher-Order Statistics for Few-Shot Class-Incremental Learning with Pre-trained Vision Transformers](https://mlsanity.com/paper/2404.06622/) (2024)
- [Revisiting Pool-based Prompt Learning for Few-shot Class-incremental Learning](https://mlsanity.com/paper/2507.09183/) (2025)
- [NTK-Guided Few-Shot Class Incremental Learning](https://mlsanity.com/paper/2403.12486/) (2024)
- [Semantic-visual Guided Transformer for Few-shot Class-incremental Learning](https://mlsanity.com/paper/2303.15494/) (2023)
- [Continual Learning: Forget-free Winning Subnetworks for Video Representations](https://mlsanity.com/paper/2312.11973/) (2023)
- [Forget-free Continual Learning with Soft-Winning SubNetworks](https://mlsanity.com/paper/2303.14962/) (2023)
- [Progressive Fourier Neural Representation for Sequential Video Compilation](https://mlsanity.com/paper/2306.11305/) (2023)
- [HyperMask: Adaptive Hypernetwork-based Masks for Continual Learning](https://mlsanity.com/paper/2310.00113/) (2023)
- [Latest Advancements Towards Catastrophic Forgetting under Data Scarcity: A Comprehensive Survey on Few-Shot Class Incremental Learning](https://mlsanity.com/paper/2502.08181/) (2025)
- [Self-Regulated Neurogenesis for Online Data-Incremental Learning](https://mlsanity.com/paper/2403.14684/) (2024)
- [Learn by Reasoning: Analogical Weight Generation for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2503.21258/) (2025)
- [Mitigating Catastrophic Forgetting in the Incremental Learning of Medical Images](https://mlsanity.com/paper/2504.20033/) (2025)
- [A Bag of Tricks for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2403.14392/) (2024)
- [I2CANSAY:Inter-Class Analogical Augmentation and Intra-Class Significance Analysis for Non-Exemplar Online Task-Free Continual Learning](https://mlsanity.com/paper/2404.13576/) (2024)
- [Covariance-based Space Regularization for Few-shot Class Incremental Learning](https://mlsanity.com/paper/2411.01172/) (2024)
- [Tripartite Weight-Space Ensemble for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2506.15720/) (2025)
- [Bias Mitigating Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2402.00481/) (2024)
- [Controllable Relation Disentanglement for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2403.11070/) (2024)
- [Consistency-Guided Asynchronous Contrastive Tuning for Few-Shot Class-Incremental Tuning of Foundation Models](https://mlsanity.com/paper/2405.16625/) (2024)
- [Unsupervised Few-Shot Continual Learning for Remote Sensing Image Scene Classification](https://mlsanity.com/paper/2406.18574/) (2024)
- [Continual Deep Learning on the Edge via Stochastic Local Competition among Subnetworks](https://mlsanity.com/paper/2407.10758/) (2024)
- [Knowledge Adaptation Network for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2409.11770/) (2024)
- [Does Prior Data Matter? Exploring Joint Training in the Context of Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2503.10003/) (2025)
- [Knowledge Transfer-Driven Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2306.10942/) (2023)
- [AggSS: An Aggregated Self-Supervised Approach for Class-Incremental Learning](https://mlsanity.com/paper/2408.04347/) (2024)

## Abstract (arXiv)

Inspired by Regularized Lottery Ticket Hypothesis (RLTH), which hypothesizes that there exist smooth (non-binary) subnetworks within a dense network that achieve the competitive performance of the dense network, we propose a few-shot class incremental learning (FSCIL) method referred to as Soft-SubNetworks (SoftNet). Our objective is to learn a sequence of sessions incrementally, where each session only includes a few training instances per class while preserving the knowledge of the previously learned ones. SoftNet jointly learns the model weights and adaptive non-binary soft masks at a base training session in which each mask consists of the major and minor subnetwork; the former aims to minimize catastrophic forgetting during training, and the latter aims to avoid overfitting to a few samples in each new training session. We provide comprehensive empirical validations demonstrating that our SoftNet effectively tackles the few-shot incremental learning problem by surpassing the performance of state-of-the-art baselines over benchmark datasets.