mlsanityData to 14 Oct 2026

Regularizing with Pseudo-Negatives for Continual Self-Supervised Learning

arXiv 2306.05101 ↗First posted 8 Jun 2023cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We introduce a novel Pseudo-Negative Regularization (PNR) framework for effective continual self-supervised learning (CSSL). Our PNR leverages pseudo-negatives obtained through model-based augmentation in a way that newly learned representations may not contradict what has been learned in the past. Specifically, for the InfoNCE-based contrastive learning methods, we define symmetric pseudo-negatives obtained from current and previous models and use them in both main and regularization loss terms. Furthermore, we extend this idea to non-contrastive learning methods which do not inherently rely on negatives. For these methods, a pseudo-negative is defined as the output from the previous model for a differently augmented version of the anchor sample and is asymmetrically applied to the regularization term. Extensive experimental results demonstrate that our PNR framework achieves state-of-the-art performance in representation learning during CSSL by effectively balancing the trade-off between plasticity and stability.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

33 of 49 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2020Supervised Contrastive Learningcited by 2,829
  8. 2016Learning without Forgettingcited by 2,496
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  11. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  12. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
21 more
  1. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  2. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  3. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  5. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  6. 2019Large Scale Incremental Learningcited by 732
  7. 2019Three scenarios for continual learningcited by 614
  8. 2017Large Batch Training of Convolutional Networkscited by 535
  9. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
  10. 2022FOSTER: Feature Boosting and Compression for Class-Incremental Learningcited by 229
  11. 2021Self-Supervised Models are Continual Learnerscited by 149
  12. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 146
  13. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113
  14. 2020Continual Learning with Node-Importance based Adaptive Group Sparse Regularizationcited by 105
  15. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  16. 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 76
  17. 2023On the Stability-Plasticity Dilemma of Class-Incremental Learningcited by 56
  18. 2021Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Frameworkcited by 49
  19. 2021How Well Does Self-Supervised Pre-Training Perform with Streaming Data?cited by 29
  20. 2023Kaizen: Practical Self-supervised Continual Learning with Continual Fine-tuningcited by 16
  21. 2022Towards Diverse Evaluation of Class Incremental Learning: A Representation Learning Perspectivecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 1.

8 papers.

  1. 2024Hyperparameters in Continual Learning: A Reality Checkcited by 12
  2. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  3. 2025Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanationcited by 9
  4. 2024Memory-efficient Continual Learning with Neural Collapse Contrastivecited by 8
  5. 2025Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learningcited by 2
  6. 2025SAMix: Calibrated and Accurate Continual Learning via Sphere-Adaptive Mixup and Neural Collapsecited by 1
  7. 2025Towards Robust Incremental Learning under Ambiguous Supervisioncited by 0
  8. 2026Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Modelscited by 0