mlsanityData to 7 Oct 2026

Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners

arXiv 2201.04924 ↗First posted 13 Jan 2022cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

In the SSLAD-Track 3B challenge on continual learning, we propose the method of COntinual Learning with Transformer (COLT). We find that transformers suffer less from catastrophic forgetting compared to convolutional neural network. The major principle of our method is to equip the transformer based feature extractor with old knowledge distillation and head expanding strategies to compete catastrophic forgetting. In this report, we first introduce the overall framework of continual learning for object detection. Then, we analyse the key elements' effect on withstanding catastrophic forgetting in our solution. Our method achieves 70.78 mAP on the SSLAD-Track 3B challenge test set.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 20 of its 22 references that are in the map. First posted on arXiv 13 Jan 2022.

Builds on

20 of 22 references.

References · 20

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2016Learning without Forgettingcited by 2,465
  3. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  5. 2017Continual Learning with Deep Generative Replaycited by 1,152
  6. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  7. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  8. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  9. 2018End-to-End Incremental Learningcited by 596
  10. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  11. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  12. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
8 more
  1. 2021Gradient Projection Memory for Continual Learningcited by 338
  2. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  3. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  4. 2021Distilling Causal Effect of Data in Class-Incremental Learningcited by 117
  5. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  6. 2020Generative Feature Replay For Class-Incremental Learningcited by 109
  7. 2020GAN Memory with No Forgettingcited by 84
  8. 2021Continual Learning via Bit-Level Information Preservingcited by 32

Built on by

Cited by year

Citing papers per year: 2022: 11; 2023: 3; 2024: 6.

20 papers.

Sort:

By area:

  1. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  2. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  3. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  4. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  5. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  6. 2022A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentialscited by 45
  7. 2022Memory Efficient Continual Learning with Transformerscited by 43
  8. 2022Continual Object Detection: A review of definitions, strategies, and challengescited by 25
  9. 2022CoMFormer: Continual Learning in Semantic and Panoptic Segmentationcited by 25
  10. 2022Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularizationcited by 23
  11. 2024Adaptive Memory Replay for Continual Learningcited by 21
  12. 2022Continual Learning with Transformers for Image Classificationcited by 17
8 more
  1. 2022Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbonescited by 12
  2. 2023Effects of Architectures on Continual Semantic Segmentationcited by 3
  3. 2023CHEEM: Continual Learning by Reuse, New, Adapt and Skip -- A Hierarchical Exploration-Exploitation Approachcited by 1
  4. 2024Sequential Learning in the Dense Associative Memorycited by 1
  5. 2022E2-AEN: End-to-End Incremental Learning with Adaptively Expandable Networkcited by 0
  6. 2022Delving into Transformer for Incremental Semantic Segmentationcited by 0
  7. 2023Dissecting Continual Learning a Structural and Data Analysiscited by 0
  8. 2024Lifelong Person Searchcited by 0