CLA: Latent Alignment for Online Continual Self-Supervised Learning
Not on the map. Why?
Abstract (arXiv)
Self-supervised learning (SSL) is able to build latent representations that generalize well to unseen data. However, only a few SSL techniques exist for the online CL setting, where data arrives in small minibatches, the model must comply with a fixed computational budget, and task boundaries are absent. We introduce Continual Latent Alignment (CLA), a novel SSL strategy for Online CL that aligns the representations learned by the current model with past representations to mitigate forgetting. We found that our CLA is able to speed up the convergence of the training process in the online scenario, outperforming state-of-the-art approaches under the same computational budget. Surprisingly, we also discovered that using CLA as a pretraining protocol in the early stages of pretraining leads to a better final performance when compared to a full i.i.d. pretraining.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 14 Jul 2025.
Builds on
41 of 54 references.
References · 41
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2020Exploring Simple Siamese Representation Learningcited by 2,866
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
- 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
29 more
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
- 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
- 2019Three scenarios for continual learningcited by 614
- 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 245
- 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 219
- 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
- 2021Self-Supervised Models are Continual Learnerscited by 149
- 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
- 2021Representational Continuity for Unsupervised Continual Learningcited by 95
- 2020Rethinking Experience Replay: a Bag of Tricks for Continual Learningcited by 94
- 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 83
- 2023Computationally Budgeted Continual Learning: What Does Matter?cited by 61
- 2021Self-Supervised Training Enhances Online Continual Learningcited by 54
- 2021Continually Learning Self-Supervised Representations with Projected Functional Regularizationcited by 38
- 2022The Challenges of Continuous Self-Supervised Learningcited by 36
- 2024Exemplar-free Continual Representation Learning via Learnable Drift Compensationcited by 31
- 2021Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffercited by 30
- 2023Online Continual Learning Without the Storage Constraintcited by 29
- 2023A Comprehensive Empirical Evaluation on Online Continual Learningcited by 24
- 2023Rethinking Momentum Knowledge Distillation in Online Continual Learningcited by 18
- 2022SCALE: Online Self-Supervised Lifelong Learning without Prior Knowledgecited by 16
- 2023Improving Online Continual Learning Performance and Stability with Temporal Ensemblescited by 14
- 2023Plasticity-Optimized Complementary Networks for Unsupervised Continual Learningcited by 14
- 2024Forgetting, Ignorance or Myopia: Revisiting Key Challenges in Online Continual Learningcited by 12
- 2021Hypernetworks for Continual Semi-Supervised Learningcited by 11
- 2025Replay-free Online Continual Learning with Self-Supervised MultiPatchescited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.