mlsanityData to 14 Oct 2026

Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners

arXiv 2404.02117 ↗First posted 2 Apr 2024cs.CVcited by 36

On the map: Knowledge distillation → Distillation for continual learning

Abstract (arXiv)

Few-Shot Class Incremental Learning (FSCIL) is a task that requires a model to learn new classes incrementally without forgetting when only a few samples for each class are given. FSCIL encounters two significant challenges: catastrophic forgetting and overfitting, and these challenges have driven prior studies to primarily rely on shallow models, such as ResNet-18. Even though their limited capacity can mitigate both forgetting and overfitting issues, it leads to inadequate knowledge transfer during few-shot incremental sessions. In this paper, we argue that large models such as vision and language transformers pre-trained on large datasets can be excellent few-shot incremental learners. To this end, we propose a novel FSCIL framework called PriViLege, Pre-trained Vision and Language transformers with prompting functions and knowledge distillation. Our framework effectively addresses the challenges of catastrophic forgetting and overfitting in large models through new pre-trained knowledge tuning (PKT) and two losses: entropy-based divergence loss and semantic knowledge distillation loss. Experimental results show that the proposed PriViLege significantly outperforms the existing state-of-the-art methods with a large margin, e.g., +9.38% in CUB200, +20.58% in CIFAR-100, and +13.36% in miniImageNet. Our implementation code is available at https://github.com/KHU-AGI/PriViLege.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Distillation for continual learning, one of 12 directions in Knowledge distillation. Builds on 32 of its 44 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

32 of 44 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2021Learning to Prompt for Continual Learningcited by 944
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  11. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  12. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
20 more
  1. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  2. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  3. 2020Few-Shot Class-Incremental Learningcited by 251
  4. 2021Few-Shot Incremental Learning with Continually Evolved Classifierscited by 187
  5. 2020A Review of Generalized Zero-Shot Learning Methodscited by 159
  6. 2022Forward Compatible Few-Shot Class-Incremental Learningcited by 127
  7. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 125
  8. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  9. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 116
  10. 2022Constrained Few-shot Class-incremental Learningcited by 94
  11. 2022Few-Shot Class-Incremental Learning by Sampling Multi-Phase Taskscited by 88
  12. 2022Few-Shot Class-Incremental Learning from an Open-Set Perspectivecited by 84
  13. 2021Few-Shot Lifelong Learningcited by 59
  14. 2021Subspace Regularizers for Few-Shot Class Incremental Learningcited by 56
  15. 2023Introducing Language Guidance in Prompt-based Continual Learningcited by 49
  16. 2023A Survey on Few-Shot Class-Incremental Learningcited by 48
  17. 2023When Prompt-based Incremental Learning Does Not Meet Strong Pretrainingcited by 41
  18. 2023Online Class Incremental Learning on Stochastic Blurry Task Boundary via Mask and Visual Prompt Tuningcited by 35
  19. 2024Learning Prompt with Distribution-Based Feature Replay for Few-Shot Class-Incremental Learningcited by 12
  20. 2023LFS-GAN: Lifelong Few-Shot Image Generationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 15; 2026: 12.

36 papers.

Sort:

By area:

  1. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  2. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 18
  3. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 13
  4. 2024Versatile Incremental Learning: Towards Class and Domain-Agnostic Incremental Learningcited by 9
  5. 2025Revisiting Pool-based Prompt Learning for Few-shot Class-incremental Learningcited by 8
  6. 2024Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learningcited by 7
  7. 2025DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformationscited by 7
  8. 2024Online Continuous Generalized Category Discoverycited by 5
  9. 2025Pay Less Attention to Deceptive Artifacts: Robust Detection of Compressed Deepfakes on Online Social Networkscited by 5
  10. 2024CFTS-GAN: Continual Few-Shot Teacher Student for Generative Adversarial Networkscited by 4
  11. 2025Latest Advancements Towards Catastrophic Forgetting under Data Scarcity: A Comprehensive Survey on Few-Shot Class Incremental Learningcited by 4
  12. 2025ProtoConNet: Prototypical Augmentation and Alignment for Open-Set Few-Shot Image Classificationcited by 3
24 more
  1. 2024Adapter-Enhanced Semantic Prompting for Continual Learningcited by 2
  2. 2025DSS-Prompt: Dynamic-Static Synergistic Prompting for Few-Shot Class-Incremental Learningcited by 2
  3. 2024Consistency-Guided Asynchronous Contrastive Tuning for Few-Shot Class-Incremental Tuning of Foundation Modelscited by 1
  4. 2024Parameter-Efficient Active Learning for Foundational modelscited by 1
  5. 2025Continuous Knowledge-Preserving Decomposition with Adaptive Layer Selection for Few-Shot Class-Incremental Learningcited by 1
  6. 2025Few-Shot Class-Incremental Model Attribution Using Learnable Representation From CLIP-ViT Featurescited by 1
  7. 2025Does Prior Data Matter? Exploring Joint Training in the Context of Few-Shot Class-Incremental Learningcited by 1
  8. 2025Beyond Synthetic Replays: Turning Diffusion Features into Few-Shot Class-Incremental Learning Knowledgecited by 1
  9. 2026Few-shot Class-Incremental Learning via Generative Co-Memory Regularizationcited by 1
  10. 2024ReFu: Recursive Fusion for Exemplar-Free 3D Class-Incremental Learningcited by 0
  11. 2025Brain-inspired analogical mixture prototypes for few-shot class-incremental learningcited by 0
  12. 2025Adaptive Additive Parameter Updates of Vision Transformers for Few-Shot Continual Learningcited by 0
  13. 2025Diffusion-Classifier Synergy: Reward-Aligned Learning via Mutual Boosting Loop for FSCILcited by 0
  14. 2025SAFA-SNN: Sparsity-Aware On-Device Few-Shot Class-Incremental Learning with Fast-Adaptive Structure of Spiking Neural Networkcited by 0
  15. 2026CASP: Few-Shot Class-Incremental Learning with CLS Token Attention Steering Promptscited by 0
  16. 2026Feature-Space Generative Models for One-Shot Class-Incremental Learningcited by 0
  17. 2026Compensating Visual Insufficiency with Stratified Language Guidance for Long-Tail Class Incremental Learningcited by 0
  18. 2026When Sensing Varies with Contexts: Context Probing for Tactile Few-Shot Class-Incremental Learningcited by 0
  19. 2026Optical-Guided Neural Collapse for SAR Few-Shot Class Incremental Learningcited by 0
  20. 2026Adapter-Based Few-Shot Continual Learning for Malicious Packet Recognitioncited by 0
  21. 2026Online Versatile Incremental Learning: Towards Class and Domain-Agnostic Adaptation at Any Timecited by 0
  22. 2026Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learningcited by 0
  23. 2026Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learningcited by 0
  24. 2026Learning to Curate What You Generate for Generalizable Few-Shot Class-Incremental Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning
  2. 2024A streamlined Approach to Multimodal Few-Shot Class Incremental Learning for Fine-Grained Datasets
  3. 2024Few Shot Class Incremental Learning using Vision-Language models
  4. 2025Breaking Forgetting: Training-Free Few-Shot Class-Incremental Learning via Conditional Diffusion
  5. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
  6. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning
  7. 2022Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay
  8. 2024On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning