mlsanityData to 14 Oct 2026

Forget-free Continual Learning with Soft-Winning SubNetworks

arXiv 2303.14962 ↗First posted 27 Mar 2023cs.LG cs.AI cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Inspired by Regularized Lottery Ticket Hypothesis (RLTH), which states that competitive smooth (non-binary) subnetworks exist within a dense network in continual learning tasks, we investigate two proposed architecture-based continual learning methods which sequentially learn and select adaptive binary- (WSN) and non-binary Soft-Subnetworks (SoftNet) for each task. WSN and SoftNet jointly learn the regularized model weights and task-adaptive non-binary masks of subnetworks associated with each task whilst attempting to select a small set of weights to be activated (winning ticket) by reusing weights of the prior subnetworks. Our proposed WSN and SoftNet are inherently immune to catastrophic forgetting as each selected subnetwork model does not infringe upon other subnetworks in Task Incremental Learning (TIL). In TIL, binary masks spawned per winning ticket are encoded into one N-bit binary digit mask, then compressed using Huffman coding for a sub-linear increase in network capacity to the number of tasks. Surprisingly, in the inference step, SoftNet generated by injecting small noises to the backgrounds of acquired WSN (holding the foregrounds of WSN) provides excellent forward transfer power for future tasks in TIL. SoftNet shows its effectiveness over WSN in regularizing parameters to tackle the overfitting, to a few examples in Few-shot Class Incremental Learning (FSCIL).

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 59 of its 82 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

59 of 82 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2021Convex optimizationcited by 2,515
  5. 2016Learning without Forgettingcited by 2,496
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  11. 2017Continual Learning with Deep Generative Replaycited by 1,158
  12. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
38 more
  1. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  2. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  3. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  4. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  5. 2019Large Scale Incremental Learningcited by 732
  6. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  7. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  8. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  10. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 379
  11. 2021Gradient Projection Memory for Continual Learningcited by 348
  12. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  13. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  14. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  15. 2020Few-Shot Class-Incremental Learningcited by 251
  16. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 237
  17. 2020Supermasks in Superpositioncited by 227
  18. 2013Predicting Parameters in Deep Learningcited by 224
  19. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 207
  20. 2017Hierarchical Representations for Efficient Architecture Searchcited by 191
  21. 2021Few-Shot Incremental Learning with Continually Evolved Classifierscited by 187
  22. 2018Reinforced Continual Learningcited by 186
  23. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  24. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
  25. 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
  26. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 123
  27. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  28. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 117
  29. 2021Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learningcited by 115
  30. 2020Continual Learning with Node-Importance based Adaptive Group Sparse Regularizationcited by 105
  31. 2018Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilizationcited by 104
  32. 2019Continual Learning via Neural Pruningcited by 101
  33. 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
  34. 2021Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learningcited by 96
  35. 2022Constrained Few-shot Class-incremental Learningcited by 94
  36. 2022Few-Shot Class-Incremental Learning by Sampling Multi-Phase Taskscited by 88
  37. 2022Few-Shot Class-Incremental Learning from an Open-Set Perspectivecited by 84
  38. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 72

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2025Continual Reinforcement Learning by Planning with Online World Modelscited by 8
  2. 2025Mitigating Catastrophic Forgetting in the Incremental Learning of Medical Imagescited by 3
  3. 2024HINT: Hypernetwork Approach to Training Weight Interval Regions in Continual Learningcited by 2
  4. 2024Fixed Random Classifier Rearrangement for Continual Learningcited by 0
  5. 2026SPARS: A Stability-Plasticity-Aware Region Selection Framework for Continual Learningcited by 0