mlsanityData to 14 Oct 2026

Compacting, Picking and Growing for Unforgetting Continual Learning

arXiv 1910.06562 ↗First posted 15 Oct 2019cs.LG stat.MLcited by 207

Not on the map. Why?

Abstract (arXiv)

Continual lifelong learning is essential to many applications. In this paper, we propose a simple but effective approach to continual deep learning. Our approach leverages the principles of deep model compression, critical weights selection, and progressive networks expansion. By enforcing their integration in an iterative manner, we introduce an incremental learning method that is scalable to the number of sequential tasks in a continual learning process. Our approach is easy to implement and owns several favorable characteristics. First, it can avoid forgetting (i.e., learn new tasks while remembering all previous tasks). Second, it allows model expansion but can maintain the model compactness when handling sequential tasks. Besides, through our compaction and selection/expansion mechanism, we show that the knowledge accumulated through learning previous tasks is helpful to build a better model for the new tasks compared to training the models independently with tasks. Experimental results show that our approach can incrementally learn a deep model tackling multiple tasks without forgetting, while the model compactness is maintained with the performance more satisfiable than individual task training.

Key facts. Cited by 207 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 20; 2022: 39; 2023: 57; 2024: 44; 2025: 34; 2026: 13.

207 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  3. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  4. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  5. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  6. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 229
  7. 2023Class-Incremental Learning: A Surveycited by 209
  8. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  9. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 143
  10. 2022Continual Learning and Private Unlearningcited by 130
  11. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 125
  12. 2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
38 more
  1. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  2. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  3. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 82
  4. 2022A Theoretical Study on Solving Continual Learningcited by 80
  5. 2024Continual Learning and Catastrophic Forgettingcited by 77
  6. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  7. 2022Class-Incremental Learning with Strong Pre-trained Modelscited by 60
  8. 2022Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfercited by 59
  9. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  10. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  11. 2021Efficient Feature Transformations for Discriminative and Generative Continual Learningcited by 50
  12. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 49
  13. 2022A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentialscited by 45
  14. 2023MiVOLO: Multi-input Transformer for Age and Gender Estimationcited by 42
  15. 2021Facial expression and attributes recognition based on multi-task learning of lightweight neural networkscited by 40
  16. 2023Continual Learning on Dynamic Graphs via Parameter Isolationcited by 36
  17. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 35
  18. 2021Continual Learning for Recurrent Neural Networks: an Empirical Evaluationcited by 34
  19. 2023Audio-Visual Class-Incremental Learningcited by 33
  20. 2023A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithmcited by 33
  21. 2024How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?cited by 33
  22. 2022Graph Lifelong Learning: A Surveycited by 30
  23. 2022Meta-attention for ViT-backed Continual Learningcited by 29
  24. 2023Learnability and Algorithm for Continual Learningcited by 28
  25. 2022Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractorcited by 27
  26. 2022How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Compositioncited by 27
  27. 2023Dynamic Residual Classifier for Class Incremental Learningcited by 26
  28. 2022Continual Object Detection: A review of definitions, strategies, and challengescited by 25
  29. 2023SIESTA: Efficient Online Continual Learning with Sleepcited by 24
  30. 2023How Efficient Are Today's Continual Learning Algorithms?cited by 24
  31. 2023Dealing with Cross-Task Class Discrimination in Online Continual Learningcited by 24
  32. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  33. 2021Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Maskingcited by 22
  34. 2023Task-Aware Information Routing from Common Representation Space in Lifelong Learningcited by 22
  35. 2023Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospectscited by 22
  36. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  37. 2021Continual Learning in Sensor-based Human Activity Recognition: an Empirical Benchmark Analysiscited by 21
  38. 2022Continual Learning Based on OOD Detection and Task Maskingcited by 21

50 most cited of 207 All 207 (JSON)