mlsanityData to 7 Oct 2026

Co2L: Contrastive Continual Learning

arXiv 2106.14413 ↗First posted 28 Jun 2021cs.LG cs.CVcited by 243

Not on the map. Why?

Abstract (arXiv)

Recent breakthroughs in self-supervised learning show that such algorithms learn visual representations that can be transferred better to unseen tasks than joint-training methods relying on task-specific supervision. In this paper, we found that the similar holds in the continual learning con-text: contrastively learned representations are more robust against the catastrophic forgetting than jointly trained representations. Based on this novel observation, we propose a rehearsal-based continual learning algorithm that focuses on continually learning and maintaining transferable representations. More specifically, the proposed scheme (1) learns representations using the contrastive learning objective, and (2) preserves learned representations using a self-supervised distillation step. We conduct extensive experimental validations under popular benchmark image classification datasets, where our method sets the new state-of-the-art performance.

Key facts. Cited by 243 papers in the map (data through 7 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 28 Jun 2021.

Builds on

36 of 46 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  12. 2020Supervised Contrastive Learningcited by 2,803
24 more
  1. 2016Learning without Forgettingcited by 2,465
  2. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  4. 2016Progressive Neural Networkscited by 1,332
  5. 2019Contrastive Multiview Codingcited by 1,297
  6. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  7. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  8. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  9. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  10. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  11. 2020Contrastive Learning with Hard Negative Samplescited by 619
  12. 2019Three scenarios for continual learningcited by 610
  13. 2019Gradient based sample selection for online continual learningcited by 568
  14. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  15. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  16. 2019Meta-Learning Representations for Continual Learningcited by 189
  17. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 160
  18. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  19. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  20. 2018Measuring and regularizing networks in function spacecited by 97
  21. 2018Self-Supervised GANs via Auxiliary Rotation Losscited by 94
  22. 2021Self-Supervised Training Enhances Online Continual Learningcited by 53
  23. 2021Reproducibility Report: La-MAML: Look-ahead Meta Learning for Continual Learningcited by 42
  24. 2020Self-Supervised Learning Aided Class-Incremental Lifelong Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 37; 2023: 74; 2024: 61; 2025: 36; 2026: 25.

243 papers.

Sort:

By area:

  1. 2021Learning to Prompt for Continual Learningcited by 938
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  4. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  5. 2023Class-Incremental Learning: A Surveycited by 209
  6. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  7. 2023Mitigating the Alignment Tax of RLHFcited by 181
  8. 2021Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Datacited by 152
  9. 2021Self-Supervised Models are Continual Learnerscited by 149
  10. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  11. 2022A Theoretical Study on Solving Continual Learningcited by 78
  12. 2022Representation Compensation Networks for Continual Semantic Segmentationcited by 72
38 more
  1. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  2. 2022CLIP model is an Efficient Continual Learnercited by 64
  3. 2023Computationally Budgeted Continual Learning: What Does Matter?cited by 61
  4. 2021Recent Advances of Continual Learning in Computer Vision: An Overviewcited by 55
  5. 2022SparCL: Sparse Continual Learning on the Edgecited by 55
  6. 2022A Simple Baseline that Questions the Use of Pretrained-Models in Continual Learningcited by 54
  7. 2023Online Prototype Learning for Online Continual Learningcited by 53
  8. 2023PCR: Proxy-based Contrastive Replay for Online Class-Incremental Continual Learningcited by 50
  9. 2023Introducing Language Guidance in Prompt-based Continual Learningcited by 49
  10. 2023AttriCLIP: A Non-Incremental Learner for Incremental Knowledge Learningcited by 46
  11. 2022Transfer without Forgettingcited by 44
  12. 2022Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interferencecited by 44
  13. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  14. 2021DualNet: Continual Learning, Fast and Slowcited by 39
  15. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  16. 2022CLAD: A realistic Continual Learning benchmark for Autonomous Drivingcited by 35
  17. 2023Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learningcited by 34
  18. 2022OpenCon: Open-world Contrastive Learningcited by 33
  19. 2022Grow and Merge: A Unified Framework for Continuous Categories Discoverycited by 33
  20. 2022A simple but strong baseline for online continual learning: Repeated Augmented Rehearsalcited by 32
  21. 2023A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithmcited by 32
  22. 2024Exemplar-free Continual Representation Learning via Learnable Drift Compensationcited by 30
  23. 2023Asynchronous Federated Continual Learningcited by 29
  24. 2024Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learningcited by 29
  25. 2023Learnability and Algorithm for Continual Learningcited by 28
  26. 2023Continual Learning with Low Rank Adaptationcited by 27
  27. 2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 26
  28. 2023Tangent Model Composition for Ensembling and Continual Fine-tuningcited by 25
  29. 2023ConSlide: Asynchronous Hierarchical Interaction Transformer with Breakup-Reorganize Rehearsal for Continual Whole Slide Image Analysiscited by 25
  30. 2023Dealing with Cross-Task Class Discrimination in Online Continual Learningcited by 24
  31. 2023Prompt-augmented Temporal Point Process for Streaming Event Sequencecited by 24
  32. 2023Continual Face Forgery Detection via Historical Distribution Preservingcited by 23
  33. 2022InCloud: Incremental Learning for Point Cloud Place Recognitioncited by 22
  34. 2023Task-Aware Information Routing from Common Representation Space in Lifelong Learningcited by 22
  35. 2023Steering Prototypes with Prompt-tuning for Rehearsal-free Continual Learningcited by 22
  36. 2023Open-World Continual Learning: Unifying Novelty Detection and Continual Learningcited by 22
  37. 2023Class-Incremental Grouping Network for Continual Audio-Visual Learningcited by 22
  38. 2022Continual Learning Based on OOD Detection and Task Maskingcited by 21

50 most cited of 243 All 243 (JSON)