mlsanityData to 7 Oct 2026

ScrollNet: Dynamic Weight Importance for Continual Learning

arXiv 2308.16567 ↗First posted 31 Aug 2023cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

The principle underlying most existing continual learning (CL) methods is to prioritize stability by penalizing changes in parameters crucial to old tasks, while allowing for plasticity in other parameters. The importance of weights for each task can be determined either explicitly through learning a task-specific mask during training (e.g., parameter isolation-based approaches) or implicitly by introducing a regularization term (e.g., regularization-based approaches). However, all these methods assume that the importance of weights for each task is unknown prior to data exposure. In this paper, we propose ScrollNet as a scrolling neural network for continual learning. ScrollNet can be seen as a dynamic network that assigns the ranking of weight importance for each task before data exposure, thus achieving a more favorable stability-plasticity tradeoff during sequential task learning by reassigning this ranking for different tasks. Additionally, we demonstrate that ScrollNet can be combined with various CL methods, including regularization-based and replay-based approaches. Experimental results on CIFAR100 and TinyImagenet datasets show the effectiveness of our proposed method. We release our code at https://github.com/FireFYF/ScrollNet.git.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 46 of its 64 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

46 of 64 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2016Learning without Forgettingcited by 2,465
  5. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  8. 2016Progressive Neural Networkscited by 1,332
  9. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  10. 2017Continual Learning with Deep Generative Replaycited by 1,152
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  12. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
34 more
  1. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  2. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  3. 2019Large Scale Incremental Learningcited by 727
  4. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  5. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  6. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  7. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  8. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  9. 2021Dynamic Neural Networks: A Surveycited by 405
  10. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  11. 2018Slimmable Neural Networkscited by 301
  12. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  13. 2019Continual learning with hypernetworkscited by 267
  14. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  15. 2020Supermasks in Superpositioncited by 223
  16. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  17. 2020Understanding the Role of Training Regimes in Continual Learningcited by 177
  18. 2020PLOP: Learning without Forgetting for Continual Semantic Segmentationcited by 177
  19. 2017Adaptive Neural Networks for Efficient Inferencecited by 161
  20. 2017Convolutional Networks with Adaptive Inference Graphscited by 147
  21. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 139
  22. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  23. 2020Generative Feature Replay For Class-Incremental Learningcited by 109
  24. 2018Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilizationcited by 104
  25. 2017Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Executioncited by 59
  26. 2021Layerwise Optimization by Gradient Decomposition for Continual Learningcited by 51
  27. 2021Zero Time Waste: Recycling Predictions in Early Exit Neural Networkscited by 39
  28. 2021Slimmable Compressive Autoencoders for Practical Neural Image Compressioncited by 38
  29. 2022CLAD: A realistic Continual Learning benchmark for Autonomous Drivingcited by 35
  30. 2020Self-Training for Class-Incremental Semantic Segmentationcited by 32
  31. 2022Balancing Stability and Plasticity through Advanced Null Space in Continual Learningcited by 29
  32. 2014Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learningcited by 24
  33. 2017Changing Model Behavior at Test-Time Using Reinforcement Learningcited by 15
  34. 2020Ternary Feature Masks: zero-forgetting for task-incremental learningcited by 14

Built on by

Cited by year

Citing papers per year: 2024: 4.

4 papers.

  1. 2024Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learningcited by 4
  2. 2024Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspectivecited by 2
  3. 2024CALA: A Class-Aware Logit Adapter for Few-Shot Class-Incremental Learningcited by 1
  4. 2024Rethinking Class-Incremental Learning from a Dynamic Imbalanced Learning Perspectivecited by 0