Investigating the Impact of Weight Sharing Decisions on Knowledge Transfer in Continual Learning
Not on the map. Why?
Abstract (arXiv)
Continual Learning (CL) has generated attention as a method of avoiding Catastrophic Forgetting (CF) in the sequential training of neural networks, improving network efficiency and adaptability to different tasks. Additionally, CL serves as an ideal setting for studying network behavior and Forward Knowledge Transfer (FKT) between tasks. Pruning methods for CL train subnetworks to handle the sequential tasks which allows us to take a structured approach to investigating FKT. Sharing prior subnetworks' weights leverages past knowledge for the current task through FKT. Understanding which weights to share is important as sharing all weights can yield sub-optimal accuracy. This paper investigates how different sharing decisions affect the FKT between tasks. Through this lens we demonstrate how task complexity and similarity influence the optimal weight sharing decisions, giving insights into the relationships between tasks and helping inform decision making in similar CL methods. We implement three sequential datasets designed to emphasize variation in task complexity and similarity, reporting results for both ResNet-18 and VGG-16. By sharing in accordance with the decisions supported by our findings, we show that we can improve task accuracy compared to other sharing decisions.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 15 of its 22 references that are in the map. First posted on arXiv 16 Nov 2023.
Builds on
15 of 22 references.
References · 15
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2017Continual Learning with Deep Generative Replaycited by 1,158
- 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
- 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
- 2018Deep Learning for Classical Japanese Literaturecited by 422
- 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
- 2020Continual Learning with Node-Importance based Adaptive Group Sparse Regularizationcited by 105
- 2019Continual Learning via Neural Pruningcited by 101
- 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 81
- 2020SpaceNet: Make Free Space For Continual Learningcited by 53
- 2023Parameter-Level Soft-Masking for Continual Learningcited by 40
- 2018Incremental Classifier Learning with Generative Adversarial Networkscited by 32
Built on by
Citing papers per year: 2024: 1.
1 paper.