mlsanityData to 7 Oct 2026

Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks

arXiv 2310.09436 ↗First posted 13 Oct 2023cs.CL cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Continual learning (CL) has two main objectives: preventing catastrophic forgetting (CF) and encouraging knowledge transfer (KT). The existing literature mainly focused on overcoming CF. Some work has also been done on KT when the tasks are similar. To our knowledge, only one method has been proposed to learn a sequence of mixed tasks. However, these techniques still suffer from CF and/or limited KT. This paper proposes a new CL method to achieve both. It overcomes CF by isolating the knowledge of each task via discovering a subnetwork for it. A soft-masking mechanism is also proposed to preserve the previous knowledge and to enable the new task to leverage the past knowledge to achieve KT. Experiments using classification, generation, information extraction, and their mixture (i.e., heterogeneous tasks) show that the proposed method consistently outperforms strong baselines.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 51 of its 65 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

51 of 65 references.

References · 51

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2017Continual Learning with Deep Generative Replaycited by 1,152
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  10. 2021Learning to Prompt for Continual Learningcited by 938
  11. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
38 more
  1. 2019Are Sixteen Heads Really Better than One?cited by 803
  2. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  3. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  4. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  5. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  6. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  7. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  8. 2020Supermasks in Superpositioncited by 223
  9. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  10. 2023Continual Pre-training of Language Modelscited by 194
  11. 2019Episodic Memory in Lifelong Language Learningcited by 188
  12. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  13. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  14. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  15. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  16. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  17. 2021Continual Learning for Text Classification with Information Disentanglement Based Regularizationcited by 86
  18. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  19. 2020Continual Learning in Task-Oriented Dialogue Systemscited by 82
  20. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 80
  21. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  22. 2022A Theoretical Study on Solving Continual Learningcited by 78
  23. 2017Deep Generative Dual Memory Network for Continual Learningcited by 59
  24. 2022Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfercited by 59
  25. 2017Continual Learning in Generative Adversarial Netscited by 51
  26. 2019Complementary Learning for Overcoming Catastrophic Forgetting Using Experience Replaycited by 51
  27. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  28. 2020Efficient Meta Lifelong-Learning with Limited Memorycited by 48
  29. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  30. 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
  31. 2022Continual Prompt Tuning for Dialog State Trackingcited by 44
  32. 2023Parameter-Level Soft-Masking for Continual Learningcited by 39
  33. 2021ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Miningcited by 37
  34. 2022Continual Sequence Generation with Adaptive Compositional Modulescited by 31
  35. 2018Large-Scale Multi-Domain Belief Tracking with Knowledge Sharingcited by 29
  36. 2023Learnability and Algorithm for Continual Learningcited by 28
  37. 2022ConTinTin: Continual Learning from Task Instructionscited by 26
  38. 2020Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Priorcited by 23

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 1.

7 papers.

  1. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  2. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  3. 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
  4. 2025Recurrent Knowledge Identification and Fusion for Language Model Continual Learningcited by 14
  5. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  6. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 7
  7. 2025AIMMerging: Adaptive Iterative Model Merging Using Training Trajectories for Language Model Continual Learningcited by 4