mlsanityData to 7 Oct 2026

Overcoming Catastrophic Forgetting by Incremental Moment Matching

arXiv 1703.08475 ↗First posted 24 Mar 2017cs.LG cs.AIcited by 243

Not on the map. Why?

Abstract (arXiv)

Catastrophic forgetting is a problem of neural networks that loses the information of the first task after training the second task. Here, we propose a method, i.e. incremental moment matching (IMM), to resolve this problem. IMM incrementally matches the moment of the posterior distribution of the neural network which is trained on the first and the second task, respectively. To make the search space of posterior parameter smooth, the IMM procedure is complemented by various transfer learning techniques including weight transfer, L2-norm of the old and the new parameter, and a variant of dropout with the old parameter. We analyze our approach on a variety of datasets including the MNIST, CIFAR-10, Caltech-UCSD-Birds, and Lifelog datasets. The experimental results show that IMM achieves state-of-the-art performance by balancing the information between an old and a new network.

Key facts. Cited by 243 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 64; 2022: 47; 2023: 55; 2024: 38; 2025: 27; 2026: 12.

243 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  5. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  6. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  7. 2023Class-Incremental Learning: A Surveycited by 209
  8. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  9. 2022Patching open-vocabulary models by interpolating weightscited by 187
  10. 2023Understanding plasticity in neural networkscited by 170
  11. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  12. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
38 more
  1. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 139
  2. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 114
  3. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  4. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  5. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 92
  6. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  7. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 80
  8. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  9. 2021Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Datacited by 67
  10. 2021Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inferencecited by 66
  11. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  12. 2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 64
  13. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  14. 2022Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfercited by 59
  15. 2021Few-Shot Lifelong Learningcited by 58
  16. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  17. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  18. 2021Model Zoo: A Growing "Brain" That Learns Continuallycited by 53
  19. 2023Online Prototype Learning for Online Continual Learningcited by 53
  20. 2024Consistent Prompting for Rehearsal-Free Continual Learningcited by 52
  21. 2021Efficient Feature Transformations for Discriminative and Generative Continual Learningcited by 50
  22. 2021Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Taskscited by 46
  23. 2021Posterior Meta-Replay for Continual Learningcited by 45
  24. 2022A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentialscited by 45
  25. 2021A Bayesian Federated Learning Framework with Online Laplace Approximationcited by 41
  26. 2023A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Taskscited by 41
  27. 2022Multi-domain Learning for Updating Face Anti-spoofing Modelscited by 39
  28. 2021Co-Transport for Class-Incremental Learningcited by 38
  29. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  30. 2021Continual Learning for Blind Image Quality Assessmentcited by 36
  31. 2021Distilled Replay: Overcoming Forgetting through Synthetic Samplescited by 36
  32. 2021Open-world Machine Learning: Applications, Challenges, and Opportunitiescited by 34
  33. 2022Grow and Merge: A Unified Framework for Continuous Categories Discoverycited by 33
  34. 2021Continual Learning via Bit-Level Information Preservingcited by 32
  35. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  36. 2021SpikeDyn: A Framework for Energy-Efficient Spiking Neural Networks with Continual and Unsupervised Learning Capabilities in Dynamic Environmentscited by 29
  37. 2021CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Taskscited by 29
  38. 2022Meta-attention for ViT-backed Continual Learningcited by 29

50 most cited of 243 All 243 (JSON)