mlsanityData to 14 Oct 2026

Self-Supervised Models are Continual Learners

arXiv 2112.04215 ↗First posted 8 Dec 2021cs.CV cs.LGcited by 149

Not on the map. Why?

Abstract (arXiv)

Self-supervised models have been shown to produce comparable or better visual representations than their supervised counterparts when trained offline on unlabeled data at scale. However, their efficacy is catastrophically reduced in a Continual Learning (CL) scenario where data is presented to the model sequentially. In this paper, we show that self-supervised loss functions can be seamlessly converted into distillation mechanisms for CL by adding a predictor network that maps the current state of the representations to their past state. This enables us to devise a framework for Continual self-supervised visual representation Learning that (i) significantly improves the quality of the learned representations, (ii) is compatible with several state-of-the-art self-supervised objectives, and (iii) needs little to no hyperparameter tuning. We demonstrate the effectiveness of our approach empirically by training six popular self-supervised models in various CL settings.

Key facts. Cited by 149 papers in the map (data through 14 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

47 of 59 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  11. 2020Supervised Contrastive Learningcited by 2,829
  12. 2016Learning without Forgettingcited by 2,496
35 more
  1. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  4. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  6. 2016Progressive Neural Networkscited by 1,361
  7. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  8. 2019Contrastive Multiview Codingcited by 1,301
  9. 2017Continual Learning with Deep Generative Replaycited by 1,158
  10. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  11. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  13. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  14. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
  15. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  16. 2019Large Scale Incremental Learningcited by 732
  17. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  18. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
  19. 2018End-to-End Incremental Learningcited by 597
  20. 2017Large Batch Training of Convolutional Networkscited by 535
  21. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  22. 2020Whitening for Self-Supervised Representation Learningcited by 279
  23. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  24. 2021Co2L: Contrastive Continual Learningcited by 244
  25. 2019Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learningcited by 182
  26. 2019Continual Unsupervised Representation Learningcited by 147
  27. 2019A continual learning survey: Defying forgetting in classification taskscited by 118
  28. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  29. 2020Memory-Efficient Incremental Learning Through Feature Adaptationcited by 107
  30. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  31. 2021Self-Supervised Training Enhances Online Continual Learningcited by 54
  32. 2020Online Continual Learning under Extreme Memory Constraintscited by 48
  33. 2018Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologiescited by 43
  34. 2019Unsupervised Progressive Learning and the STAM Architecturecited by 30
  35. 2021SPeCiaL: Self-Supervised Pretraining for Continual Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 16; 2023: 45; 2024: 41; 2025: 26; 2026: 21.

149 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  2. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  3. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 180
  4. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  5. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
  6. 2022Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replaycited by 72
  7. 2023Continual Learning: Applications and the Road Forwardcited by 59
  8. 2023Online Prototype Learning for Online Continual Learningcited by 53
  9. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  10. 2023Test-Time Training on Video Streamscited by 37
  11. 2023Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learningcited by 34
  12. 2023Audio-Visual Class-Incremental Learningcited by 33
38 more
  1. 2023Downstream-agnostic Adversarial Examplescited by 32
  2. 2024Exemplar-free Continual Representation Learning via Learnable Drift Compensationcited by 31
  3. 2023ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysiscited by 25
  4. 2023A Comprehensive Empirical Evaluation on Online Continual Learningcited by 24
  5. 2022Continual Barlow Twins: continual self-supervised learning for remote sensing semantic segmentationcited by 22
  6. 2024Securely Fine-tuning Pre-trained Encoders Against Adversarial Examplescited by 21
  7. 2022Challenging Common Assumptions about Catastrophic Forgettingcited by 20
  8. 2023Semi-supervised learning made simple with self-supervised clusteringcited by 20
  9. 2023EMP-SSL: Towards Self-Supervised Learning in One Training Epochcited by 19
  10. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 19
  11. 2023Class Incremental Learning with Self-Supervised Pre-Training and Prototype Learningcited by 18
  12. 2022A soft nearest-neighbor framework for continual semi-supervised learningcited by 17
  13. 2023PromptFusion: Decoupling Stability and Plasticity for Continual Learningcited by 17
  14. 2022Consistency is the key to further mitigating catastrophic forgetting in continual learningcited by 16
  15. 2022SCALE: Online Self-Supervised Lifelong Learning without Prior Knowledgecited by 16
  16. 2023Kaizen: Practical Self-supervised Continual Learning with Continual Fine-tuningcited by 16
  17. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  18. 2025Task-Agnostic Guided Feature Expansion for Class-Incremental Learningcited by 15
  19. 2023On the Effectiveness of LayerNorm Tuning for Continual Learning in Vision Transformerscited by 14
  20. 2023Plasticity-Optimized Complementary Networks for Unsupervised Continual Learningcited by 14
  21. 2024Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network Structurecited by 13
  22. 2024Provable Contrastive Continual Learningcited by 13
  23. 2025Componential Prompt-Knowledge Alignment for Domain Incremental Learningcited by 13
  24. 2024Hyperparameters in Continual Learning: A Reality Checkcited by 12
  25. 2022Beyond Supervised Continual Learning: a Reviewcited by 11
  26. 2023Large-scale Pre-trained Models are Surprisingly Strong in Incremental Novel Class Discoverycited by 11
  27. 2024Random Representations Outperform Online Continually Learned Representationscited by 11
  28. 2024Contrastive Continual Learning with Importance Sampling and Prototype-Instance Relation Distillationcited by 11
  29. 2024CP-Prompt: Composition-Based Cross-modal Prompting for Domain-Incremental Continual Learningcited by 11
  30. 2024Semi-supervised Open-World Object Detectioncited by 10
  31. 2024SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-trainingcited by 10
  32. 2024Queryable Prototype Multiple Instance Learning with Vision-Language Models for Incremental Whole Slide Image Classificationcited by 10
  33. 2024ViLCo-Bench: VIdeo Language COntinual learning Benchmarkcited by 9
  34. 2022Learning Representations for New Sound Classes With Continual Self-Supervised Learningcited by 8
  35. 2023Regularizing with Pseudo-Negatives for Continual Self-Supervised Learningcited by 8
  36. 2024Integrating Present and Past in Unsupervised Continual Learningcited by 8
  37. 2024PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learningcited by 8
  38. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 8

50 most cited of 149 All 149 (JSON)