mlsanityData to 7 Oct 2026

Continual Semi-Supervised Learning through Contrastive Interpolation Consistency

arXiv 2108.06552 ↗First posted 14 Aug 2021stat.ML cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Continual Learning (CL) investigates how to train Deep Networks on a stream of tasks without incurring forgetting. CL settings proposed in literature assume that every incoming example is paired with ground-truth annotations. However, this clashes with many real-world applications: gathering labeled data, which is in itself tedious and expensive, becomes infeasible when data flow as a stream. This work explores Continual Semi-Supervised Learning (CSSL): here, only a small fraction of labeled input examples are shown to the learner. We assess how current CL methods (e.g.: EWC, LwF, iCaRL, ER, GDumb, DER) perform in this novel and challenging scenario, where overfitting entangles forgetting. Subsequently, we design a novel CSSL method that exploits metric learning and consistency regularization to leverage unlabeled examples while learning. We show that our proposal exhibits higher resilience to diminishing supervision and, even more surprisingly, relying only on 25% supervision suffices to outperform SOTA methods trained under full supervision.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 14 Aug 2021.

Builds on

37 of 52 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2016Learning without Forgettingcited by 2,465
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  7. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  10. 2016Progressive Neural Networkscited by 1,332
  11. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  12. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
25 more
  1. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  2. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  3. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  4. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  5. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  6. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  7. 2019Gradient based sample selection for online continual learningcited by 568
  8. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  9. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  10. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  11. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  12. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  13. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  14. 2017Unlabeled Samples Generated by GAN Improve the Person Re-identification Baseline in vitrocited by 280
  15. 2018Task-Free Continual Learningcited by 219
  16. 2017Incremental Learning of Object Detectors without Catastrophic Forgettingcited by 208
  17. 2019Class-incremental Learning via Deep Model Consolidationcited by 172
  18. 2020Using Hindsight to Anchor Past Knowledge in Continual Learningcited by 160
  19. 2018Towards Robust Evaluations of Continual Learningcited by 148
  20. 2020Conditional Channel Gated Networks for Task-Aware Continual Learningcited by 115
  21. 2018Measuring and regularizing networks in function spacecited by 97
  22. 2020Rethinking Experience Replay: a Bag of Tricks for Continual Learningcited by 94
  23. 2019Task Agnostic Continual Learning via Meta Learningcited by 53
  24. 2020Meta-Consolidation for Continual Learningcited by 40
  25. 2020Lifelong Object Detectioncited by 25

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 5; 2024: 12; 2025: 3.

25 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2022Transfer without Forgettingcited by 44
  3. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  4. 2024AIDE: An Automatic Data Engine for Object Detection in Autonomous Drivingcited by 26
  5. 2022A soft nearest-neighbor framework for continual semi-supervised learningcited by 17
  6. 2023Plasticity-Optimized Complementary Networks for Unsupervised Continual Learningcited by 14
  7. 2022Consistency-based Self-supervised Learning for Temporal Anomaly Localizationcited by 10
  8. 2023Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learningcited by 8
  9. 2025SacFL: Self-Adaptive Federated Continual Learning for Resource-Constrained End Devicescited by 8
  10. 2023Latent Spectral Regularization for Continual Learningcited by 6
  11. 2024Joint Diffusion models in Continual Learningcited by 6
  12. 2023Category Adaptation Meets Projected Distillation in Generalized Continual Category Discoverycited by 5
13 more
  1. 2022Contrastive Learning for Online Semi-Supervised General Continual Learningcited by 4
  2. 2024Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computationcited by 4
  3. 2024IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learningcited by 4
  4. 2025Divide-and-Conquer for Enhancing Unlabeled Learning, Stability, and Plasticity in Semi-supervised Continual Learningcited by 3
  5. 2024Sequential Visual and Semantic Consistency for Semi-supervised Text Recognitioncited by 2
  6. 2024Mitigating Interference in the Knowledge Continuum through Attention-Guided Incremental Learningcited by 2
  7. 2024TACLE: Task and Class-aware Exemplar-free Semi-supervised Class Incremental Learningcited by 1
  8. 2024CUAL: Continual Uncertainty-aware Active Learnercited by 1
  9. 2024Uncertainty Quantification in Continual Open-World Learningcited by 1
  10. 2024Pragmatist Intelligence: Where the Principle of Usefulness Can Take ANNscited by 0
  11. 2024AggSS: An Aggregated Self-Supervised Approach for Class-Incremental Learningcited by 0
  12. 2024CONCLAD: COntinuous Novel CLAss Detectorcited by 0
  13. 2025Towards Robust Incremental Learning under Ambiguous Supervisioncited by 0