mlsanityData to 7 Oct 2026

Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima

arXiv 2111.01549 ↗First posted 30 Oct 2021cs.LG cs.CVcited by 118

Not on the map. Why?

Abstract (arXiv)

This paper considers incremental few-shot learning, which requires a model to continually recognize new categories with only a few examples provided. Our study shows that existing methods severely suffer from catastrophic forgetting, a well-known problem in incremental learning, which is aggravated due to data scarcity and imbalance in the few-shot setting. Our analysis further suggests that to prevent catastrophic forgetting, actions need to be taken in the primitive stage -- the training of base classes instead of later few-shot learning sessions. Therefore, we propose to search for flat local minima of the base training objective function and then fine-tune the model parameters within the flat region on new tasks. In this way, the model can efficiently learn new classes while preserving the old ones. Comprehensive experimental results demonstrate that our approach outperforms all prior state-of-the-art methods and is very close to the approximate upper bound. The source code is available at https://github.com/moukamisama/F2M.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 43 of its 71 references that are in the map. First posted on arXiv 30 Oct 2021.

Builds on

43 of 71 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  5. 2016Learning without Forgettingcited by 2,465
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  7. 2016Matching Networks for One Shot Learningcited by 2,246
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  9. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  10. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  11. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  12. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
31 more
  1. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  2. 2018End-to-End Incremental Learningcited by 596
  3. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  4. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  6. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  7. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  8. 2018Meta-Transfer Learning for Few-Shot Learningcited by 306
  9. 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
  10. 2018Learning without Memorizingcited by 249
  11. 2020Few-Shot Class-Incremental Learningcited by 249
  12. 2020Semantic Drift Compensation for Class-Incremental Learningcited by 241
  13. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  14. 2019Cross Attention Network for Few-shot Classificationcited by 218
  15. 2021Few-Shot Incremental Learning with Continually Evolved Classifierscited by 184
  16. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  17. 2018Task-Agnostic Meta-Learning for Few-shot Learningcited by 151
  18. 2020Adaptive Aggregation Networks for Class-Incremental Learningcited by 149
  19. 2021Distilling Causal Effect of Data in Class-Incremental Learningcited by 117
  20. 2021Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learningcited by 114
  21. 2020Interventional Few-Shot Learningcited by 112
  22. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  23. 2020iTAML: An Incremental Task-Agnostic Meta-learning Approachcited by 90
  24. 2018Incremental Few-Shot Learning with Attention Attractor Networkscited by 65
  25. 2021Few-Shot Lifelong Learningcited by 58
  26. 2020DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learningcited by 49
  27. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  28. 2019An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learningcited by 38
  29. 2019Learning Adaptive Classifiers Synthesis for Generalized Few-Shot Learningcited by 29
  30. 2020Entropic gradient descent algorithms and wide flat minimacited by 25
  31. 2019Meta-Transfer Learning through Hard Taskscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 33; 2024: 35; 2025: 23; 2026: 12.

118 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2023Class-Incremental Learning: A Surveycited by 209
  3. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  4. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  5. 2022Constrained Few-shot Class-incremental Learningcited by 93
  6. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  7. 2023Learning with Fantasy: Semantic-Aware Virtual Contrastive Constraint for Few-Shot Class-Incremental Learningcited by 55
  8. 2023A Survey on Few-Shot Class-Incremental Learningcited by 48
  9. 2022CoSCL: Cooperation of Small Continual Learners is Stronger than a Big Onecited by 43
  10. 2022On the Soft-Subnetwork for Few-shot Class Incremental Learningcited by 37
  11. 2023Continual Learning on Dynamic Graphs via Parameter Isolationcited by 35
  12. 2024Make Continual Learning Stronger via C-Flatcited by 35
38 more
  1. 2024Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learnerscited by 35
  2. 2022Memorizing Complementation Network for Few-Shot Class-Incremental Learningcited by 30
  3. 2023Active Neural Mappingcited by 30
  4. 2024Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learningcited by 26
  5. 2023Few-shot Class-incremental Learning for Classification and Object Detection: A Surveycited by 25
  6. 2023S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learningcited by 23
  7. 2024Generative Multi-modal Models are Good Class-Incremental Learnerscited by 22
  8. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  9. 2023Learning Expressive Prompting With Residuals for Vision Transformerscited by 21
  10. 2022Momentum-based Weight Interpolation of Strong Zero-Shot Models for Continual Learningcited by 19
  11. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  12. 2023Reinforcement Unlearningcited by 16
  13. 2024Few-Shot Class Incremental Learning with Attention-Aware Self-Adaptive Promptcited by 15
  14. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  15. 2024Rethinking Few-shot Class-incremental Learning: Learning from Yourselfcited by 13
  16. 2024Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learningcited by 12
  17. 2023LFS-GAN: Lifelong Few-Shot Image Generationcited by 11
  18. 2024Training Machine Learning models at the Edge: A Surveycited by 11
  19. 2024CLOSER: Towards Better Representation Learning for Few-Shot Class-Incremental Learningcited by 11
  20. 2023Gaussian Switch Sampling: A Second Order Approach to Active Learningcited by 10
  21. 2023Wide Flat Minimum Watermarking for Robust Ownership Verification of GANscited by 10
  22. 2022In-memory Realization of In-situ Few-shot Continual Learning with a Dynamically Evolving Explicit Memorycited by 9
  23. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  24. 2025Advancing Prompt-Based Methods for Replay-Independent General Continual Learningcited by 9
  25. 2026SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuningcited by 8
  26. 2022Semi-supervised Drifted Stream Learning with Short Lookbackcited by 7
  27. 2022Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greekcited by 7
  28. 2024Contrastive Augmented Graph2Graph Memory Interaction for Few Shot Continual Learningcited by 7
  29. 2024NTK-Guided Few-Shot Class Incremental Learningcited by 7
  30. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  31. 2023Semantic-visual Guided Transformer for Few-shot Class-incremental Learningcited by 6
  32. 2024Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minimacited by 6
  33. 2024Delta: A Cloud-assisted Data Enrichment Framework for On-Device Continual Learningcited by 6
  34. 2023Generalized Few-Shot Continual Learning with Contrastive Mixture of Adapterscited by 5
  35. 2023Improved Region Proposal Network for Enhanced Few-Shot Object Detectioncited by 5
  36. 2023Advancing continual lifelong learning in neural information retrieval: definition, dataset, framework, and empirical evaluationcited by 5
  37. 2023Continual Learning: Forget-free Winning Subnetworks for Video Representationscited by 5
  38. 2024Towards Few-Shot Learning in the Open World: A Review and Beyondcited by 5

50 most cited of 118 All 118 (JSON)