mlsanityData to 14 Oct 2026

Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning

arXiv 2103.04059 ↗First posted 6 Mar 2021cs.CVcited by 115

On the map: Knowledge distillation → Distillation for continual learning

Abstract (arXiv)

Few-shot class incremental learning (FSCIL) portrays the problem of learning new concepts gradually, where only a few examples per concept are available to the learner. Due to the limited number of examples for training, the techniques developed for standard incremental learning cannot be applied verbatim to FSCIL. In this work, we introduce a distillation algorithm to address the problem of FSCIL and propose to make use of semantic information during training. To this end, we make use of word embeddings as semantic information which is cheap to obtain and which facilitate the distillation process. Furthermore, we propose a method based on an attention mechanism on multiple parallel embeddings of visual data to align visual and semantic vectors, which reduces issues related to catastrophic forgetting. Via experiments on MiniImageNet, CUB200, and CIFAR100 dataset, we establish new state-of-the-art results by outperforming existing approaches.

Key facts. Cited by 115 papers in the map (data through 14 Oct 2026). Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation. Builds on 27 of its 42 references that are in the map. First posted on arXiv 6 Mar 2021.

Builds on

27 of 42 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2016Learning without Forgettingcited by 2,496
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  7. 2016Matching Networks for One Shot Learningcited by 2,255
  8. 2017Continual Learning with Deep Generative Replaycited by 1,158
  9. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  10. 2019Large Scale Incremental Learningcited by 732
  11. 2019Three scenarios for continual learningcited by 614
  12. 2018End-to-End Incremental Learningcited by 597
15 more
  1. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
  2. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  3. 2017Variational Continual Learningcited by 339
  4. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
  5. 2020Few-Shot Class-Incremental Learningcited by 251
  6. 2017Incremental Learning of Object Detectors without Catastrophic Forgettingcited by 208
  7. 2019Class-incremental Learning via Deep Model Consolidationcited by 173
  8. 2016An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wildcited by 146
  9. 2016Latent Embeddings for Zero-shot Classificationcited by 129
  10. 2018Incremental Few-Shot Learning with Attention Attractor Networkscited by 66
  11. 2019Transductive Zero-Shot Learning for 3D Point Cloud Classificationcited by 39
  12. 2019Zero-shot Learning of 3D Point Cloud Objectscited by 34
  13. 2020XtarNet: Learning to Extract Task-Adaptive Representation for Incremental Few-Shot Learningcited by 31
  14. 2019Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objectscited by 24
  15. 2020Any-Shot Object Detectioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 22; 2023: 35; 2024: 25; 2025: 13; 2026: 7.

115 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  3. 2022Forward Compatible Few-Shot Class-Incremental Learningcited by 127
  4. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  5. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 116
  6. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  7. 2022Constrained Few-shot Class-incremental Learningcited by 94
  8. 2022Few-Shot Class-Incremental Learning by Sampling Multi-Phase Taskscited by 88
  9. 2021On Learning the Geodesic Path for Incremental Learningcited by 85
  10. 2022Few-Shot Class-Incremental Learning from an Open-Set Perspectivecited by 84
  11. 2022Representation Compensation Networks for Continual Semantic Segmentationcited by 72
  12. 2021Subspace Regularizers for Few-Shot Class Incremental Learningcited by 56
38 more
  1. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  2. 2023Learning with Fantasy: Semantic-Aware Virtual Contrastive Constraint for Few-Shot Class-Incremental Learningcited by 55
  3. 2023A Survey on Few-Shot Class-Incremental Learningcited by 48
  4. 2023Few-Shot Class-Incremental Learning via Training-Free Prototype Calibrationcited by 46
  5. 2021Co-Transport for Class-Incremental Learningcited by 38
  6. 2021Continually Learning Self-Supervised Representations with Projected Functional Regularizationcited by 38
  7. 2022On the Soft-Subnetwork for Few-shot Class Incremental Learningcited by 37
  8. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  9. 2022Learning with Recoverable Forgettingcited by 30
  10. 2022Memorizing Complementation Network for Few-Shot Class-Incremental Learningcited by 30
  11. 2024OrCo: Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learningcited by 26
  12. 2023Few-shot Class-incremental Learning for Classification and Object Detection: A Surveycited by 25
  13. 2022Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketchescited by 23
  14. 2023S3C: Self-Supervised Stochastic Classifiers for Few-Shot Class-Incremental Learningcited by 23
  15. 2022Geometer: Graph Few-Shot Class-Incremental Learning via Prototype Representationcited by 22
  16. 2024Generative Multi-modal Models are Good Class-Incremental Learnerscited by 22
  17. 2022SphereFed: Hyperspherical Federated Learningcited by 19
  18. 2023Multimodal Parameter-Efficient Few-Shot Class Incremental Learningcited by 19
  19. 2021Reinforced Attention for Few-Shot Learning and Beyondcited by 18
  20. 2022Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representationcited by 16
  21. 2023Uncertainty-Aware Distillation for Semi-Supervised Few-Shot Class-Incremental Learningcited by 16
  22. 2023New Insights for the Stability-Plasticity Dilemma in Online Continual Learningcited by 16
  23. 2023Exploring Data Geometry for Continual Learningcited by 15
  24. 2023Image-free Classifier Injection for Zero-Shot Classificationcited by 15
  25. 2024Few-Shot Class Incremental Learning with Attention-Aware Self-Adaptive Promptcited by 15
  26. 2023Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variantscited by 14
  27. 2023Plasticity-Optimized Complementary Networks for Unsupervised Continual Learningcited by 14
  28. 2024Rethinking Few-shot Class-incremental Learning: Learning from Yourselfcited by 14
  29. 2022Few-shot Class-incremental Learning for 3D Point Cloud Objectscited by 13
  30. 2023Non-Exemplar Online Class-incremental Continual Learning via Dual-prototype Self-augment and Refinementcited by 13
  31. 2022Rethinking Few-Shot Class-Incremental Learning with Open-Set Hypothesis in Hyperbolic Geometrycited by 12
  32. 2023Subspace Distillation for Continual Learningcited by 12
  33. 2023Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memorycited by 12
  34. 2023When Meta-Learning Meets Online and Continual Learning: A Surveycited by 12
  35. 2023Continual Learning for Image Segmentation with Dynamic Querycited by 12
  36. 2024Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learningcited by 12
  37. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  38. 2021Towards Data-Free Domain Generalizationcited by 11

50 most cited of 115 All 115 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners
  2. 2024On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning
  3. 2026Learning to Curate What You Generate for Generalizable Few-Shot Class-Incremental Learning
  4. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation
  5. 2023Few-shot Class-Incremental Semantic Segmentation via Pseudo-Labeling and Knowledge Distillation
  6. 2024Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning
  7. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning
  8. 2021Incremental Meta-Learning via Episodic Replay Distillation for Few-Shot Image Recognition