mlsanityData to 7 Oct 2026

S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning

arXiv 2207.12819 ↗First posted 26 Jul 2022cs.CV cs.LGcited by 246

Not on the map. Why?

Cited by 28 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

State-of-the-art deep neural networks are still struggling to address the catastrophic forgetting problem in continual learning. In this paper, we propose one simple paradigm (named as S-Prompting) and two concrete approaches to highly reduce the forgetting degree in one of the most typical continual learning scenarios, i.e., domain increment learning (DIL). The key idea of the paradigm is to learn prompts independently across domains with pre-trained transformers, avoiding the use of exemplars that commonly appear in conventional methods. This results in a win-win game where the prompting can achieve the best for each domain. The independent prompting across domains only requests one single cross-entropy loss for training and one simple K-NN operation as a domain identifier for inference. The learning paradigm derives an image prompt learning approach and a novel language-image prompt learning approach. Owning an excellent scalability (0.03% parameter increase per domain), the best of our approaches achieves a remarkable relative improvement (an average of about 30%) over the best of the state-of-the-art exemplar-free methods for three standard DIL tasks, and even surpasses the best of them relatively by about 6% in average when they use exemplars. Source code is available at urlhttps://github.com/iamwangyabin/S-Prompts.

Key facts. Cited by 246 papers in the map (data through 7 Oct 2026). Builds on 68 of its 81 references that are in the map. First posted on arXiv 26 Jul 2022.

Builds on

68 of 81 references.

References · 68

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2020Supervised Contrastive Learningcited by 2,803
  11. 2016Learning without Forgettingcited by 2,465
  12. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
38 more
  1. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  5. 2022Visual Prompt Tuningcited by 1,674
  6. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  7. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  8. 2017Continual Learning with Deep Generative Replaycited by 1,152
  9. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  10. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  11. 2021Learning to Prompt for Continual Learningcited by 938
  12. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  13. 2019Large Scale Incremental Learningcited by 727
  14. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  15. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  16. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  17. 2018End-to-End Incremental Learningcited by 596
  18. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  19. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  20. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  21. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  22. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  23. 2021Gradient Projection Memory for Continual Learningcited by 338
  24. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  25. 2019On Tiny Episodic Memories in Continual Learningcited by 299
  26. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  27. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
  28. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  29. 2021Co2L: Contrastive Continual Learningcited by 243
  30. 2020Semantic Drift Compensation for Class-Incremental Learningcited by 241
  31. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  32. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  33. 2018Continual Learning of Context-dependent Processing in Neural Networkscited by 178
  34. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  35. 2021Self-Supervised Models are Continual Learnerscited by 149
  36. 2021Training Networks in Null Space of Feature Covariance for Continual Learningcited by 139
  37. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125
  38. 2021Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learningcited by 114

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 46; 2024: 75; 2025: 75; 2026: 46.

246 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  3. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  4. 2023Class-Incremental Learning: A Surveycited by 209
  5. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  6. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  7. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 162
  8. 2024Continual Learning with Pre-Trained Models: A Surveycited by 151
  9. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  10. 2023Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimalitycited by 121
  11. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  12. 2023Learning without Forgetting for Vision-Language Modelscited by 79
38 more
  1. 2023Freestyle Layout-to-Image Synthesiscited by 67
  2. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  3. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  4. 2023Introducing Language Guidance in Prompt-based Continual Learningcited by 49
  5. 2024Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusioncited by 47
  6. 2023AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learningcited by 46
  7. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  8. 2022Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interferencecited by 44
  9. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  10. 2023When Prompt-based Incremental Learning Does Not Meet Strong Pretrainingcited by 41
  11. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 41
  12. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  13. 2023Progressive Visual Prompt Learning with Contrastive Feature Re-formationcited by 35
  14. 2023Visual Tuningcited by 34
  15. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Modelscited by 34
  16. 2023A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithmcited by 32
  17. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  18. 2024Visual Prompt Tuning in Null Space for Continual Learningcited by 30
  19. 2024Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learningcited by 29
  20. 2023Continual Learning with Low Rank Adaptationcited by 27
  21. 2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 26
  22. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  23. 2023Overcoming Generic Knowledge Loss with Selective Parameter Updatecited by 23
  24. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  25. 2023Steering Prototypes with Prompt-tuning for Rehearsal-free Continual Learningcited by 22
  26. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Modelscited by 21
  27. 2022ConStruct-VL: Data-Free Continual Structured VL Concepts Learningcited by 20
  28. 2024OVOR: OnePrompt with Virtual Outlier Regularization for Rehearsal-Free Class-Incremental Learningcited by 20
  29. 2024HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuningcited by 20
  30. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  31. 2025Continual Multimodal Contrastive Learningcited by 19
  32. 2025Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learningcited by 19
  33. 2023Class Incremental Learning with Self-Supervised Pre-Training and Prototype Learningcited by 18
  34. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  35. 2023`A-la-carte Prompt Tuning (APT): Combining Distinct Data Via Composable Promptingcited by 17
  36. 2023PromptFusion: Decoupling Stability and Plasticity for Continual Learningcited by 17
  37. 2024SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Modelscited by 17
  38. 2024Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learningcited by 16

50 most cited of 246 All 246 (JSON)