mlsanityData to 7 Oct 2026

Self-Supervised Training Enhances Online Continual Learning

arXiv 2103.14010 ↗First posted 25 Mar 2021cs.CVcited by 53

Not on the map. Why?

Abstract (arXiv)

In continual learning, a system must incrementally learn from a non-stationary data stream without catastrophic forgetting. Recently, multiple methods have been devised for incrementally learning classes on large-scale image classification tasks, such as ImageNet. State-of-the-art continual learning methods use an initial supervised pre-training phase, in which the first 10% - 50% of the classes in a dataset are used to learn representations in an offline manner before continual learning of new classes begins. We hypothesize that self-supervised pre-training could yield features that generalize better than supervised learning, especially when the number of samples used for pre-training is small. We test this hypothesis using the self-supervised MoCo-V2, Barlow Twins, and SwAV algorithms. On ImageNet, we find that these methods outperform supervised pre-training considerably for online continual learning, and the gains are larger when fewer samples are available. Our findings are consistent across three online continual learning algorithms. Our best system achieves a 14.95% relative increase in top-1 accuracy on class incremental ImageNet over the prior state of the art for online continual learning.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 60 of its 84 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

60 of 84 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2016Learning without Forgettingcited by 2,465
  10. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  11. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
38 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  4. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  5. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  9. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  10. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  11. 2016Colorful Image Colorizationcited by 1,113
  12. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  13. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  14. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  15. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  16. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  17. 2019Large Scale Incremental Learningcited by 727
  18. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  19. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  20. 2018End-to-End Incremental Learningcited by 596
  21. 2019Selective Kernel Networkscited by 401
  22. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 369
  23. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  24. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  25. 2018Learning without Memorizingcited by 249
  26. 2020Few-Shot Class-Incremental Learningcited by 249
  27. 2014Towards Open World Recognitioncited by 226
  28. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  29. 2017Incremental Learning of Object Detectors without Catastrophic Forgettingcited by 208
  30. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  31. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  32. 2020Modeling the Background for Incremental Learning in Semantic Segmentationcited by 189
  33. 2017FearNet: Brain-Inspired Model for Incremental Learningcited by 177
  34. 2019Class-incremental Learning via Deep Model Consolidationcited by 172
  35. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  36. 2019Incremental Learning Techniques for Semantic Segmentationcited by 141
  37. 2018Memory Efficient Experience Replay for Streaming Learningcited by 128
  38. 2019A continual learning survey: Defying forgetting in classification taskscited by 118

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 14; 2023: 20; 2024: 10; 2025: 4; 2026: 1.

53 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2021Co2L: Contrastive Continual Learningcited by 243
  3. 2021Self-Supervised Models are Continual Learnerscited by 149
  4. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  5. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  6. 2022A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learningcited by 54
  7. 2023Online Prototype Learning for Online Continual Learningcited by 53
  8. 2022Online Continual Learning for Embedded Devicescited by 48
  9. 2022CLAD: A realistic Continual Learning benchmark for Autonomous Drivingcited by 35
  10. 2023A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithmcited by 32
  11. 2024Exemplar-free Continual Representation Learning via Learnable Drift Compensationcited by 30
  12. 2022Continual Object Detection: A review of definitions, strategies, and challengescited by 25
38 more
  1. 2023SIESTA: Efficient Online Continual Learning with Sleepcited by 24
  2. 2023MaskCon: Masked Contrastive Learning for Coarse-Labelled Datasetcited by 21
  3. 2022Self-Supervision Can Be a Good Few-Shot Learnercited by 19
  4. 2023Kaizen: Practical Self-supervised Continual Learning with Continual Fine-tuningcited by 16
  5. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 16
  6. 2023Plasticity-Optimized Complementary Networks for Unsupervised Continual Learningcited by 14
  7. 2022Task Agnostic Representation Consolidation: a Self-supervised based Continual Learning Approachcited by 13
  8. 2023Overcoming the Stability Gap in Continual Learningcited by 13
  9. 2023GRASP: A Rehearsal Policy for Efficient Online Continual Learningcited by 13
  10. 2024Object Detectors in the Open Environment: Challenges, Solutions, and Outlookcited by 13
  11. 2024Provable Contrastive Continual Learningcited by 13
  12. 2022Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2cited by 11
  13. 2021Self-Supervised Class Incremental Learningcited by 8
  14. 2022Learning Representations for New Sound Classes With Continual Self-Supervised Learningcited by 8
  15. 2024COPR: Continual Human Preference Learning via Optimal Policy Regularizationcited by 8
  16. 2024Integrating Present and Past in Unsupervised Continual Learningcited by 8
  17. 2023Partial Hypernetworks for Continual Learningcited by 7
  18. 2023An Analysis of Initial Training Strategies for Exemplar-Free Class-Incremental Learningcited by 7
  19. 2024Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applicationscited by 7
  20. 2023Online Continual Learning via Logit Adjusted Softmaxcited by 5
  21. 2023From Categories to Classifiers: Name-Only Continual Learning by Exploring the Webcited by 5
  22. 2024Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learningcited by 5
  23. 2021Online Unsupervised Learning of Visual Representations and Categoriescited by 4
  24. 2022Causes of Catastrophic Forgetting in Class-Incremental Semantic Segmentationcited by 4
  25. 2023Principles of Forgetting in Domain-Incremental Semantic Segmentation in Adverse Weather Conditionscited by 4
  26. 2022Cold Start Streaming Learning for Deep Networkscited by 3
  27. 2022System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Gamescited by 3
  28. 2023Adiabatic replay for continual learningcited by 3
  29. 2024Balancing Continual Learning and Fine-tuning for Human Activity Recognitioncited by 3
  30. 2022Mitigating Forgetting in Online Continual Learning via Contrasting Semantically Distinct Augmentationscited by 2
  31. 2023Are Labels Needed for Incremental Instance Learning?cited by 2
  32. 2023Towards Label-Efficient Incremental Learning: A Surveycited by 2
  33. 2023COPR: Continual Learning Human Preference through Optimal Policy Regularizationcited by 2
  34. 2024Improving Data-aware and Parameter-aware Robustness for Continual Learningcited by 2
  35. 2025Exploring Incremental Unlearning: Techniques, Challenges, and Future Directionscited by 2
  36. 2025CLDyB: Towards Dynamic Benchmarking for Continual Learning with Pre-trained Modelscited by 2
  37. 2025CLA: Latent Alignment for Online Continual Self-Supervised Learningcited by 1
  38. 2023Towards Robust Feature Learning with t-vFM Similarity for Continual Learningcited by 0

50 most cited of 53 All 53 (JSON)