Continual Learners are Incremental Model Generalizers
Not on the map. Why?
Abstract (arXiv)
Motivated by the efficiency and rapid convergence of pre-trained models for solving downstream tasks, this paper extensively studies the impact of Continual Learning (CL) models as pre-trainers. In both supervised and unsupervised CL, we find that the transfer quality of the representation often increases gradually without noticeable degradation in fine-tuning performance. This is because CL models can learn improved task-general features when easily forgetting task-specific knowledge. Based on this observation, we suggest a new unsupervised CL framework with masked modeling, which aims to capture fluent task-generic representation during training. Furthermore, we propose a new fine-tuning scheme, GLobal Attention Discretization (GLAD), that preserves rich task-generic representation during solving downstream tasks. The model fine-tuned with GLAD achieves competitive performance and can also be used as a good pre-trained model itself. We believe this paper breaks the barriers between pre-training and fine-tuning steps and leads to a sustainable learning framework in which the continual learner incrementally improves model generalization, yielding better transfer to unseen tasks.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 21 Jun 2023.
Builds on
42 of 54 references.
References · 42
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2020Exploring Simple Siamese Representation Learningcited by 2,866
- 2016Learning without Forgettingcited by 2,496
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
- 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
30 more
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
- 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
- 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
- 2019Large Scale Incremental Learningcited by 732
- 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
- 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
- 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 521
- 2019Online Continual Learning with Maximally Interfered Retrievalcited by 379
- 2019On Tiny Episodic Memories in Continual Learningcited by 307
- 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
- 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
- 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
- 2019Maintaining Discrimination and Fairness in Class Incremental Learningcited by 283
- 2021Self-Supervised Models are Continual Learnerscited by 149
- 2019Continual Unsupervised Representation Learningcited by 147
- 2019Scalable and Order-robust Continual Learning with Additive Parameter Decompositioncited by 125
- 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 123
- 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 117
- 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
- 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
- 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
- 2019Rehearsal-Free Continual Learning over Small Non-I.I.D. Batchescited by 46
- 2023Is forgetting less a good inductive bias for forward transfer?cited by 18
- 2020Generative Feature Replay with Orthogonal Weight Modification for Continual Learningcited by 10
Built on by
Citing papers per year: 2023: 2; 2024: 0; 2025: 1.
3 papers.