mlsanityData to 7 Oct 2026

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

arXiv 2111.11326 ↗First posted 22 Nov 2021cs.CV cs.LGcited by 291

Not on the map. Why?

Cited by 27 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks. A recent trend indicates that dynamic architectures based on an expansion of the parameters can reduce catastrophic forgetting efficiently in continual learning. However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks. As a result, they struggle to scale to a large number of tasks without significant overhead. In this paper, we propose a transformer architecture based on a dedicated encoder/decoder framework. Critically, the encoder and decoder are shared among all tasks. Through a dynamic expansion of special tokens, we specialize each forward of our decoder network on a task distribution. Our strategy scales to a large number of tasks while having negligible memory and time overheads due to strict control of the parameters expansion. Moreover, this efficient strategy doesn't need any hyperparameter tuning to control the network's expansion. Our model reaches excellent results on CIFAR100 and state-of-the-art performances on the large-scale ImageNet100 and ImageNet1000 while having less parameters than concurrent dynamic frameworks.

Key facts. Cited by 291 papers in the map (data through 7 Oct 2026). Builds on 66 of its 86 references that are in the map. First posted on arXiv 22 Nov 2021.

Builds on

66 of 86 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  11. 2016Layer Normalizationcited by 5,579
  12. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
38 more
  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2016Learning without Forgettingcited by 2,465
  7. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  8. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2016Progressive Neural Networkscited by 1,332
  12. 2017Random Erasing Data Augmentationcited by 1,235
  13. 2017Continual Learning with Deep Generative Replaycited by 1,152
  14. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  15. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  16. 2021Going deeper with Image Transformerscited by 759
  17. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  18. 2019Large Scale Incremental Learningcited by 727
  19. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  20. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  21. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  22. 2018End-to-End Incremental Learningcited by 596
  23. 2017Learning multiple visual domains with residual adapterscited by 553
  24. 2017Convolutional Sequence to Sequence Learningcited by 550
  25. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  26. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  27. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  28. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  29. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  30. 2021Gradient Projection Memory for Continual Learningcited by 338
  31. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  32. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  33. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  34. 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 281
  35. 2018Learning without Memorizingcited by 249
  36. 2019Compacting, Picking and Growing for Unforgetting Continual Learningcited by 207
  37. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  38. 2020Modeling the Background for Incremental Learning in Semantic Segmentationcited by 189

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 28; 2023: 72; 2024: 80; 2025: 69; 2026: 41.

291 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  3. 2024What matters when building vision-language models?cited by 321
  4. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  5. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 228
  6. 2023Class-Incremental Learning: A Surveycited by 209
  7. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  8. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  9. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  10. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  11. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  12. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
38 more
  1. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  2. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  3. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 71
  4. 2022CLIP model is an Efficient Continual Learnercited by 64
  5. 2023PILOT: A Pre-Trained Model-Based Continual Learning Toolboxcited by 64
  6. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  7. 2023No One Left Behind: Real-World Federated Class-Incremental Learningcited by 53
  8. 2024Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusioncited by 47
  9. 2023Create Your World: Lifelong Text-to-Image Diffusioncited by 46
  10. 2022A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentialscited by 45
  11. 2023Dense Network Expansion for Class Incremental Learningcited by 45
  12. 2022Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interferencecited by 44
  13. 2022Memory Efficient Continual Learning with Transformerscited by 43
  14. 2023When Prompt-based Incremental Learning Does Not Meet Strong Pretrainingcited by 41
  15. 2021Continually Learning Self-Supervised Representations with Projected Functional Regularizationcited by 38
  16. 2022CLAD: A realistic Continual Learning benchmark for Autonomous Drivingcited by 35
  17. 2023The Ideal Continual Learner: An Agent That Never Forgetscited by 34
  18. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Modelscited by 34
  19. 2023Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detectioncited by 32
  20. 2024Convolutional Prompting meets Language Models for Continual Learningcited by 32
  21. 2024How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?cited by 32
  22. 2024Exemplar-free Continual Representation Learning via Learnable Drift Compensationcited by 30
  23. 2024Continual Forgetting for Pre-trained Vision Modelscited by 29
  24. 2024Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learningcited by 29
  25. 2023Continual Learning with Low Rank Adaptationcited by 27
  26. 2022Continual Learning with Bayesian Model based on a Fixed Pre-trained Feature Extractorcited by 26
  27. 2023Dynamic Residual Classifier for Class Incremental Learningcited by 26
  28. 2022CoMFormer: Continual Learning in Semantic and Panoptic Segmentationcited by 25
  29. 2023BiRT: Bio-inspired Replay in Vision Transformers for Continual Learningcited by 25
  30. 2023ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysiscited by 25
  31. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
  32. 2023SIESTA: Efficient Online Continual Learning with Sleepcited by 24
  33. 2023How Efficient Are Today's Continual Learning Algorithms?cited by 24
  34. 2023SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformerscited by 24
  35. 2023Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomycited by 24
  36. 2022Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularizationcited by 23
  37. 2022Modeling Missing Annotations for Incremental Learning in Object Detectioncited by 23
  38. 2023Steering Prototypes with Prompt-tuning for Rehearsal-free Continual Learningcited by 22

50 most cited of 291 All 291 (JSON)