mlsanityData to 7 Oct 2026

SPeCiaL: Self-Supervised Pretraining for Continual Learning

arXiv 2106.09065 ↗First posted 16 Jun 2021cs.CV cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper presents SPeCiaL: a method for unsupervised pretraining of representations tailored for continual learning. Our approach devises a meta-learning objective that differentiates through a sequential learning process. Specifically, we train a linear model over the representations to match different augmented views of the same image together, each view presented sequentially. The linear model is then evaluated on both its ability to classify images it just saw, and also on images from previous iterations. This gives rise to representations that favor quick knowledge retention with minimal forgetting. We evaluate SPeCiaL in the Continual Few-Shot Learning setting, and show that it can match or outperform other supervised pretraining approaches.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

32 of 46 references.

References · 32

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2016Matching Networks for One Shot Learningcited by 2,246
  9. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  10. 2018Group Normalizationcited by 1,514
  11. 2017Continual Learning with Deep Generative Replaycited by 1,152
  12. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
20 more
  1. 2019Gradient based sample selection for online continual learningcited by 568
  2. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  3. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  4. 2017Variational Continual Learningcited by 335
  5. 2019On Tiny Episodic Memories in Continual Learningcited by 299
  6. 2019Meta-Learning Representations for Continual Learningcited by 189
  7. 2019Continual Unsupervised Representation Learningcited by 147
  8. 2018Generative replay with feedback connections as a general strategy for continual learningcited by 103
  9. 2020Learning to Continually Learncited by 82
  10. 2018Unsupervised Learning via Meta-Learningcited by 81
  11. 2018Unsupervised Meta-Learning For Few-Shot Image Classificationcited by 74
  12. 2019Micro-Batch Training with Batch-Channel Normalization and Weight Standardizationcited by 45
  13. 2018Selfless Sequential Learningcited by 44
  14. 2018Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologiescited by 43
  15. 2021Reproducibility Report: La-MAML: Look-ahead Meta Learning for Continual Learningcited by 42
  16. 2020Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learningcited by 38
  17. 2019Unsupervised Progressive Learning and the STAM Architecturecited by 29
  18. 2019Assume, Augment and Learn: Unsupervised Few-Shot Meta-Learning via Random Labels and Data Augmentationcited by 24
  19. 2018The Utility of Sparse Representations for Control in Reinforcement Learningcited by 23
  20. 2021Generalising via Meta-Examples for Continual Learning in the Wildcited by 1

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 4; 2023: 3; 2024: 3.

12 papers.

  1. 2021Self-Supervised Models are Continual Learnerscited by 149
  2. 2021Continual Learning via Local Module Compositioncited by 61
  3. 2022Continual Learning with Foundation Models: An Empirical Study of Latent Replaycited by 61
  4. 2024Exemplar-free Continual Representation Learning via Learnable Drift Compensationcited by 30
  5. 2022Continual Object Detection: A review of definitions, strategies, and challengescited by 25
  6. 2022Continual Barlow Twins: continual self-supervised learning for remote sensing semantic segmentationcited by 22
  7. 2023Kaizen: Practical Self-supervised Continual Learning with Continual Fine-tuningcited by 16
  8. 2023Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgettingcited by 16
  9. 2022Beyond Supervised Continual Learning: a Reviewcited by 11
  10. 2024Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computationcited by 4
  11. 2024Balancing Continual Learning and Fine-tuning for Human Activity Recognitioncited by 3
  12. 2023Are Labels Needed for Incremental Instance Learning?cited by 2