mlsanityData to 14 Oct 2026

Learning without Memorizing

arXiv 1811.08051 ↗First posted 20 Nov 2018cs.CV cs.LGcited by 250

Not on the map. Why?

Abstract (arXiv)

Incremental learning (IL) is an important task aimed at increasing the capability of a trained model, in terms of the number of classes recognizable by the model. The key problem in this task is the requirement of storing data (e.g. images) associated with existing classes, while teaching the classifier to learn new classes. However, this is impractical as it increases the memory requirement at every incremental step, which makes it impossible to implement IL algorithms on edge devices with limited memory. Hence, we propose a novel approach, called `Learning without Memorizing (LwM)', to preserve the information about existing (base) classes, without storing any of their data, while making the classifier progressively learn the new classes. In LwM, we present an information preserving penalty: Attention Distillation Loss (LAD), and demonstrate that penalizing the changes in classifiers' attention maps helps to retain information of the base classes, as new classes are added. We show that adding LAD to the distillation loss which is an existing information preserving loss consistently outperforms the state-of-the-art performance in the iILSVRC-small and iCIFAR-100 datasets in terms of the overall accuracy of base and incrementally learned classes.

Key facts. Cited by 250 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 49; 2022: 47; 2023: 67; 2024: 41; 2025: 31; 2026: 15.

250 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  3. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
  4. 2023Class-Incremental Learning: A Surveycited by 209
  5. 2021Few-Shot Incremental Learning with Continually Evolved Classifierscited by 187
  6. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 184
  7. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 180
  8. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  9. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 143
  10. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  11. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  12. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113
38 more
  1. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  2. 2021Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representationscited by 96
  3. 2023FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learningcited by 87
  4. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 82
  5. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 81
  6. 2022A Theoretical Study on Solving Continual Learningcited by 80
  7. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 72
  8. 2022Representation Compensation Networks for Continual Semantic Segmentationcited by 72
  9. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  10. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  11. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  12. 2021Self-Supervised Training Enhances Online Continual Learningcited by 54
  13. 2023PCR: Proxy-based Contrastive Replay for Online Class-Incremental Continual Learningcited by 52
  14. 2021Layerwise Optimization by Gradient Decomposition for Continual Learningcited by 51
  15. 2023Class-Incremental Exemplar Compression for Class-Incremental Learningcited by 50
  16. 2022Online Continual Learning for Embedded Devicescited by 49
  17. 2022Class-Incremental Learning for Action Recognition in Videoscited by 46
  18. 2022Class-incremental Novel Class Discoverycited by 42
  19. 2023Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learningcited by 41
  20. 2021Incremental Learning in Semantic Segmentation from Image Labelscited by 39
  21. 2022Multi-domain Learning for Updating Face Anti-spoofing Modelscited by 39
  22. 2023Continual Semantic Segmentation with Automatic Memory Sample Selectioncited by 39
  23. 2021Continually Learning Self-Supervised Representations with Projected Functional Regularizationcited by 38
  24. 2022Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentationcited by 37
  25. 2025Model Adaptation: Unsupervised Domain Adaptation without Source Datacited by 34
  26. 2023Audio-Visual Class-Incremental Learningcited by 33
  27. 2023A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithmcited by 33
  28. 2024Resurrecting Old Classes with New Data for Exemplar-Free Continual Learningcited by 33
  29. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 33
  30. 2023Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detectioncited by 32
  31. 2021Dual-Teacher Class-Incremental Learning With Data-Free Generative Replaycited by 31
  32. 2023Continual Learning with Low Rank Adaptationcited by 29
  33. 2022vCLIMB: A Novel Video Class Incremental Learning Benchmarkcited by 28
  34. 2022Decomposed Knowledge Distillation for Class-Incremental Semantic Segmentationcited by 28
  35. 2023Learnability and Algorithm for Continual Learningcited by 28
  36. 2022RBC: Rectifying the Biased Context in Continual Semantic Segmentationcited by 27
  37. 2022Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractorcited by 27
  38. 2022A Multi-Head Model for Continual Learning via Out-of-Distribution Replaycited by 27

50 most cited of 250 All 250 (JSON)