# SCALE: Online Self-Supervised Lifelong Learning without Prior Knowledge

arXiv: https://arxiv.org/abs/2208.11266 · HTML: https://mlsanity.com/paper/2208.11266/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 96 references that are in the map.
- First posted on arXiv 24 Aug 2022.

## Builds on (most cited first, 30 of 62)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Whitening for Self-Supervised Representation Learning](https://mlsanity.com/paper/2007.06346/) (2020)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)

## Built on by (most cited first, 16 of 16)

- [Audio-Visual Class-Incremental Learning](https://mlsanity.com/paper/2308.11073/) (2023)
- [ViLCo-Bench: VIdeo Language COntinual learning Benchmark](https://mlsanity.com/paper/2406.13123/) (2024)
- [Memory Storyboard: Leveraging Temporal Segmentation for Streaming Self-Supervised Learning from Egocentric Videos](https://mlsanity.com/paper/2501.12254/) (2025)
- [Label Delay in Online Continual Learning](https://mlsanity.com/paper/2312.00923/) (2023)
- [Overcoming Domain Drift in Online Continual Learning](https://mlsanity.com/paper/2405.09133/) (2024)
- [The impact of model size on catastrophic forgetting in Online Continual Learning](https://mlsanity.com/paper/2407.00176/) (2024)
- [Robust Emotion Recognition via Bi-Level Self-Supervised Continual Learning](https://mlsanity.com/paper/2505.10575/) (2025)
- [Domain-Aware Augmentations for Unsupervised Online General Continual Learning](https://mlsanity.com/paper/2309.06896/) (2023)
- [Neuro-mimetic Task-free Unsupervised Online Learning with Continual Self-Organizing Maps](https://mlsanity.com/paper/2402.12465/) (2024)
- [Forward-Backward Knowledge Distillation for Continual Clustering](https://mlsanity.com/paper/2405.19234/) (2024)
- [CLA: Latent Alignment for Online Continual Self-Supervised Learning](https://mlsanity.com/paper/2507.10434/) (2025)
- [Patch-Based Contrastive Learning and Memory Consolidation for Online Unsupervised Continual Learning](https://mlsanity.com/paper/2409.16391/) (2024)
- [Caption, Create, Continue: Continual Learning with Pre-trained Generative Vision-Language Models](https://mlsanity.com/paper/2409.17806/) (2024)
- [Preventing Latent Rehearsal Decay in Online Continual SSL with SOLAR](https://mlsanity.com/paper/2604.10586/) (2026)
- [Unsupervised Continual Clustering via Forward-Backward Knowledge Distillation](https://mlsanity.com/paper/2606.07474/) (2026)
- [DRDN: Decoupled Representation Dynamic Network for From-Scratch ViT Class-Incremental Learning](https://mlsanity.com/paper/2607.01630/) (2026)

## Abstract (arXiv)

Unsupervised lifelong learning refers to the ability to learn over time while memorizing previous patterns without supervision. Although great progress has been made in this direction, existing work often assumes strong prior knowledge about the incoming data (e.g., knowing the class boundaries), which can be impossible to obtain in complex and unpredictable environments. In this paper, motivated by real-world scenarios, we propose a more practical problem setting called online self-supervised lifelong learning without prior knowledge. The proposed setting is challenging due to the non-iid and single-pass data, the absence of external supervision, and no prior knowledge. To address the challenges, we propose Self-Supervised ContrAstive Lifelong LEarning without Prior Knowledge (SCALE) which can extract and memorize representations on the fly purely from the data continuum. SCALE is designed around three major components: a pseudo-supervised contrastive loss, a self-supervised forgetting loss, and an online memory update for uniform subset selection. All three components are designed to work collaboratively to maximize learning performance. We perform comprehensive experiments of SCALE under iid and four non-iid data streams. The results show that SCALE outperforms the state-of-the-art algorithm in all settings with improvements up to 3.83%, 2.77% and 5.86% in terms of kNN accuracy on CIFAR-10, CIFAR-100, and TinyImageNet datasets.