# ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning

arXiv: https://arxiv.org/abs/2101.00407 · HTML: https://mlsanity.com/paper/2101.00407/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 2 Jan 2021.

## Builds on (most cited first, 30 of 30)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Label Propagation for Deep Semi-supervised Learning](https://mlsanity.com/paper/1904.04717/) (2019)
- [Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2004.10956/) (2020)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)
- [Meta-Learning Representations for Continual Learning](https://mlsanity.com/paper/1905.12588/) (2019)
- [Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning](https://mlsanity.com/paper/1904.03137/) (2019)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [Generative Feature Replay For Class-Incremental Learning](https://mlsanity.com/paper/2004.09199/) (2020)
- [Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks](https://mlsanity.com/paper/1511.06390/) (2015)
- [There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average](https://mlsanity.com/paper/1806.05594/) (2018)
- [Memory Replay GANs: learning to generate images from new categories without forgetting](https://mlsanity.com/paper/1809.02058/) (2018)
- [Triple Generative Adversarial Nets](https://mlsanity.com/paper/1703.02291/) (2017)
- [Smooth Neighbors on Teacher Graphs for Semi-supervised Learning](https://mlsanity.com/paper/1711.00258/) (2017)
- [Triple Generative Adversarial Networks](https://mlsanity.com/paper/1912.09784/) (2019)
- [Cognitively-Inspired Model for Incremental Learning Using a Few Examples](https://mlsanity.com/paper/2002.12411/) (2020)

## Built on by (most cited first, 30 of 68)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model](https://mlsanity.com/paper/2303.05118/) (2023)
- [Hierarchical Decomposition of Prompt-Based Continual Learning: Rethinking Obscured Sub-optimality](https://mlsanity.com/paper/2310.07234/) (2023)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [Memory Replay with Data Compression for Continual Learning](https://mlsanity.com/paper/2202.06592/) (2022)
- [AFEC: Active Forgetting of Negative Transfer in Continual Learning](https://mlsanity.com/paper/2110.12187/) (2021)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [A Probabilistic Framework for Lifelong Test-Time Adaptation](https://mlsanity.com/paper/2212.09713/) (2022)
- [CoSCL: Cooperation of Small Continual Learners is Stronger than a Big One](https://mlsanity.com/paper/2207.06543/) (2022)
- [Incorporating Neuro-Inspired Adaptability for Continual Learning in Artificial Intelligence](https://mlsanity.com/paper/2308.14991/) (2023)
- [Weighted Ensemble Models Are Strong Continual Learners](https://mlsanity.com/paper/2312.08977/) (2023)
- [Continual Vision-Language Representation Learning with Off-Diagonal Information](https://mlsanity.com/paper/2305.07437/) (2023)
- [Continually Learning Self-Supervised Representations with Projected Functional Regularization](https://mlsanity.com/paper/2112.15022/) (2021)
- [Class-Incremental Learning using Diffusion Model for Distillation and Replay](https://mlsanity.com/paper/2306.17560/) (2023)
- [How Well Does Self-Supervised Pre-Training Perform with Streaming Data?](https://mlsanity.com/paper/2104.12081/) (2021)
- [AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2403.17373/) (2024)
- [ICICLE: Interpretable Class Incremental Continual Learning](https://mlsanity.com/paper/2303.07811/) (2023)
- [Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning](https://mlsanity.com/paper/2409.01128/) (2024)
- [InCloud: Incremental Learning for Point Cloud Place Recognition](https://mlsanity.com/paper/2203.00807/) (2022)
- [Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data](https://mlsanity.com/paper/2204.08932/) (2022)
- [Generative Negative Text Replay for Continual Vision-Language Pretraining](https://mlsanity.com/paper/2210.17322/) (2022)
- [A soft nearest-neighbor framework for continual semi-supervised learning](https://mlsanity.com/paper/2212.05102/) (2022)
- [Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation](https://mlsanity.com/paper/2404.00417/) (2024)
- [A Survey on Semi-Supervised Learning for Delayed Partially Labelled Data Streams](https://mlsanity.com/paper/2106.09170/) (2021)
- [Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions](https://mlsanity.com/paper/2209.01501/) (2022)
- [Plasticity-Optimized Complementary Networks for Unsupervised Continual Learning](https://mlsanity.com/paper/2309.06086/) (2023)
- [SLCA++: Unleash the Power of Sequential Fine-tuning for Continual Learning with Pre-training](https://mlsanity.com/paper/2408.08295/) (2024)

## Abstract (arXiv)

Continual learning usually assumes the incoming data are fully labeled, which might not be applicable in real applications. In this work, we consider semi-supervised continual learning (SSCL) that incrementally learns from partially labeled data. Observing that existing continual learning methods lack the ability to continually exploit the unlabeled data, we propose deep Online Replay with Discriminator Consistency (ORDisCo) to interdependently learn a classifier with a conditional generative adversarial network (GAN), which continually passes the learned data distribution to the classifier. In particular, ORDisCo replays data sampled from the conditional generator to the classifier in an online manner, exploiting unlabeled data in a time- and storage-efficient way. Further, to explicitly overcome the catastrophic forgetting of unlabeled data, we selectively stabilize parameters of the discriminator that are important for discriminating the pairs of old unlabeled data and their pseudo-labels predicted by the classifier. We extensively evaluate ORDisCo on various semi-supervised learning benchmark datasets for SSCL, and show that ORDisCo achieves significant performance improvement on SVHN, CIFAR10 and Tiny-ImageNet, compared to strong baselines.