mlsanityData to 7 Oct 2026

Continually Learning Self-Supervised Representations with Projected Functional Regularization

arXiv 2112.15022 ↗First posted 30 Dec 2021cs.CVcited by 38

Not on the map. Why?

Abstract (arXiv)

Recent self-supervised learning methods are able to learn high-quality image representations and are closing the gap with supervised approaches. However, these methods are unable to acquire new knowledge incrementally -- they are, in fact, mostly used only as a pre-training phase over IID data. In this work we investigate self-supervised methods in continual learning regimes without any replay mechanism. We show that naive functional regularization, also known as feature distillation, leads to lower plasticity and limits continual learning performance. Instead, we propose Projected Functional Regularization in which a separate temporal projection network ensures that the newly learned feature space preserves information of the previous one, while at the same time allowing for the learning of new features. This prevents forgetting while maintaining the plasticity of the learner. Comparison with other incremental learning approaches applied to self-supervision demonstrates that our method obtains competitive performance in different scenarios and on multiple datasets.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 52 of its 75 references that are in the map. First posted on arXiv 30 Dec 2021.

Builds on

52 of 75 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2016Learning without Forgettingcited by 2,465
  11. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
38 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  4. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  5. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  6. 2016Progressive Neural Networkscited by 1,332
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  8. 2017Continual Learning with Deep Generative Replaycited by 1,152
  9. 2016Colorful Image Colorizationcited by 1,113
  10. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  11. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  12. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  13. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  14. 2019Large Scale Incremental Learningcited by 727
  15. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  16. 2018End-to-End Incremental Learningcited by 596
  17. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  18. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  19. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  20. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  21. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  22. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  23. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  24. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  25. 2018Learning without Memorizingcited by 249
  26. 2020Semantic Drift Compensation for Class-Incremental Learningcited by 241
  27. 2018Task-Free Continual Learningcited by 219
  28. 2019Continual Unsupervised Representation Learningcited by 147
  29. 2018Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgettingcited by 127
  30. 2019Functional Regularisation for Continual Learning with Gaussian Processescited by 121
  31. 2021Distilling Causal Effect of Data in Class-Incremental Learningcited by 117
  32. 2021Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learningcited by 114
  33. 2020Generative Feature Replay For Class-Incremental Learningcited by 109
  34. 2017Incremental Learning Through Deep Adaptationcited by 108
  35. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 104
  36. 2018Memory Replay GANs: learning to generate images from new categories without forgettingcited by 76
  37. 2021ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learningcited by 68
  38. 2021Layerwise Optimization by Gradient Decomposition for Continual Learningcited by 51

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 8; 2025: 8; 2026: 8.

38 papers.

Sort:

  1. 2024Continual Learning and Catastrophic Forgettingcited by 73
  2. 2024Exemplar-free Continual Representation Learning via Learnable Drift Compensationcited by 30
  3. 2025Language Guided Concept Bottleneck Models for Interpretable Continual Learningcited by 28
  4. 2023Plasticity-Optimized Complementary Networks for Unsupervised Continual Learningcited by 14
  5. 2024ViLCo-Bench: VIdeo Language COntinual learning Benchmarkcited by 9
  6. 2024Integrating Present and Past in Unsupervised Continual Learningcited by 8
  7. 2022Is Multi-Task Learning an Upper Bound for Continual Learning?cited by 6
  8. 2023Preserving Linear Separability in Continual Learning by Backward Feature Projectioncited by 6
  9. 2025Memory Storyboard: Leveraging Temporal Segmentation for Streaming Self-Supervised Learning from Egocentric Videoscited by 6
  10. 2023Efficient Self-supervised Continual Learning with Progressive Task-correlated Layer Freezingcited by 5
  11. 2023Category Adaptation Meets Projected Distillation in Generalized Continual Category Discoverycited by 5
  12. 2023Revisiting Supervision for Continual Representation Learningcited by 5
26 more
  1. 2022Exemplar-free Continual Learning of Vision Transformers via Gated Class-Attention and Cascaded Feature Drift Compensationcited by 4
  2. 2024Branch-Tuning: Balancing Stability and Plasticity for Continual Self-Supervised Learningcited by 4
  3. 2024Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computationcited by 4
  4. 2023Projected Latent Distillation for Data-Agnostic Consolidation in Distributed Continual Learningcited by 3
  5. 2025Replay-free Online Continual Learning with Self-Supervised MultiPatchescited by 3
  6. 2023Are Labels Needed for Incremental Instance Learning?cited by 2
  7. 2023Towards Label-Efficient Incremental Learning: A Surveycited by 2
  8. 2025Flashbacks to Harmonize Stability and Plasticity in Continual Learningcited by 2
  9. 2022Positive Pair Distillation Considered Harmful: Continual Meta Metric Learning for Lifelong Object Re-Identificationcited by 1
  10. 2023Density Map Distillation for Incremental Object Countingcited by 1
  11. 2023Self-supervised Activity Representation Learning with Incremental Data: An Empirical Studycited by 1
  12. 2024Class incremental learning with probability dampening and cascaded gated classifiercited by 1
  13. 2025Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learningcited by 1
  14. 2025Continual Self-Supervised Learning with Masked Autoencoders in Remote Sensingcited by 1
  15. 2025CLA: Latent Alignment for Online Continual Self-Supervised Learningcited by 1
  16. 2025Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learningcited by 1
  17. 2023CSSL-MHTR: Continual Self-Supervised Learning for Scalable Multi-script Handwritten Text Recognitioncited by 0
  18. 2024Data Stream Sampling with Fuzzy Task Boundaries and Noisy Labelscited by 0
  19. 2026Continual Learning for non-stationary regression via Memory-Efficient Replaycited by 0
  20. 2026Preventing Latent Rehearsal Decay in Online Continual SSL with SOLARcited by 0
  21. 2026MoRe: Modular Representations for Principled Continual Representation Learning on Sequential Datacited by 0
  22. 2026Continual Learning as a Multiphase Moving-Boundary Problemcited by 0
  23. 2026Continual Visual and Verbal Learning Through a Child's Egocentric Inputcited by 0
  24. 2026SAOT: Self-Supervised Continual Graph Learning with Structure-Aware Optimal Transportcited by 0
  25. 2026Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Modelscited by 0
  26. 2026MAGIC-SSCIL: Manifold Anchoring and Geometric Incremental Calibration for Semi-Supervised Class Incremental Learningcited by 0