mlsanityData to 7 Oct 2026

Improving Vision Transformers for Incremental Learning

arXiv 2112.06103 ↗First posted 12 Dec 2021cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

This paper proposes a working recipe of using Vision Transformer (ViT) in class incremental learning. Although this recipe only combines existing techniques, developing the combination is not trivial. Firstly, naive application of ViT to replace convolutional neural networks (CNNs) in incremental learning results in serious performance degradation. Secondly, we nail down three issues of naively using ViT: (a) ViT has very slow convergence when the number of classes is small, (b) more bias towards new classes is observed in ViT than CNN-based architectures, and (c) the conventional learning rate of ViT is too low to learn a good classifier layer. Finally, our solution, named ViTIL (ViT for Incremental Learning) achieves new state-of-the-art on both CIFAR and ImageNet datasets for all three class incremental learning setups by a clear margin. We believe this advances the knowledge of transformer in the incremental learning community. Code will be publicly released.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 12 Dec 2021.

Builds on

34 of 45 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2016Learning without Forgettingcited by 2,465
  11. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  12. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
22 more
  1. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  2. 2017Continual Learning with Deep Generative Replaycited by 1,152
  3. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  4. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  5. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  6. 2019Large Scale Incremental Learningcited by 727
  7. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  8. 2018End-to-End Incremental Learningcited by 596
  9. 2021Early Convolutions Help Transformers See Bettercited by 395
  10. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  11. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  12. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  13. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 281
  14. 2018Learning without Memorizingcited by 249
  15. 2020Mnemonics Training: Multi-Class Incremental Learning without Forgettingcited by 220
  16. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  17. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 144
  18. 2020SS-IL: Separated Softmax for Incremental Learningcited by 126
  19. 2021Distilling Causal Effect of Data in Class-Incremental Learningcited by 117
  20. 2019Overcoming Catastrophic Forgetting with Unlabeled Data in the Wildcited by 111
  21. 2017Encoder Based Lifelong Learningcited by 102
  22. 2020ScaIL: Classifier Weights Scaling for Class Incremental Learningcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 3; 2024: 2; 2025: 1; 2026: 1.

17 papers.

Sort:

  1. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  2. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  3. 2022A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentialscited by 45
  4. 2022CLAD: A realistic Continual Learning benchmark for Autonomous Drivingcited by 35
  5. 2023BiRT: Bio-inspired Replay in Vision Transformers for Continual Learningcited by 25
  6. 2022Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularizationcited by 23
  7. 2022Self-Promoted Supervision for Few-Shot Transformercited by 22
  8. 2022Online Continual Learning with Contrastive Vision Transformercited by 21
  9. 2022Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbonescited by 12
  10. 2022D3Former: Debiased Dual Distilled Transformer for Incremental Learningcited by 10
  11. 2024CEAT: Continual Expansion and Absorption Transformer for Non-Exemplar Class-Incremental Learningcited by 5
  12. 2022Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learningcited by 3
5 more
  1. 2024ConSept: Continual Semantic Segmentation via Adapter-based Vision Transformercited by 3
  2. 2025Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learningcited by 3
  3. 2023CHEEM: Continual Learning by Reuse, New, Adapt and Skip -- A Hierarchical Exploration-Exploitation Approachcited by 1
  4. 2023Dissecting Continual Learning a Structural and Data Analysiscited by 0
  5. 2026Shortcut Solutions Learned by Transformers Impair Continual Compositional Reasoningcited by 0