# Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild

arXiv: https://arxiv.org/abs/1903.12648 · HTML: https://mlsanity.com/paper/1903.12648/

## Key facts

- Cited by 111 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2019.

## Built on by (most cited first, 30 of 111)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Online Continual Learning in Image Classification: An Empirical Survey](https://mlsanity.com/paper/2101.10423/) (2021)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Training Networks in Null Space of Feature Covariance for Continual Learning](https://mlsanity.com/paper/2103.07113/) (2021)
- [Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning](https://mlsanity.com/paper/2103.13885/) (2021)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [Distilling Causal Effect of Data in Class-Incremental Learning](https://mlsanity.com/paper/2103.01737/) (2021)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning](https://mlsanity.com/paper/2501.13198/) (2025)
- [R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learning](https://mlsanity.com/paper/2203.13104/) (2022)
- [Essentials for Class Incremental Learning](https://mlsanity.com/paper/2102.09517/) (2021)
- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [A Closer Look at Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2203.17269/) (2022)
- [Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning](https://mlsanity.com/paper/2403.18886/) (2024)
- [Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](https://mlsanity.com/paper/2208.11296/) (2022)
- [Layerwise Optimization by Gradient Decomposition for Continual Learning](https://mlsanity.com/paper/2105.07561/) (2021)
- [When Prompt-based Incremental Learning Does Not Meet Strong Pretraining](https://mlsanity.com/paper/2308.10445/) (2023)
- [Online Continual Learning For Visual Food Classification](https://mlsanity.com/paper/2108.06781/) (2021)
- [Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connector](https://mlsanity.com/paper/2110.07905/) (2021)
- [A Survey on Knowledge Editing of Neural Networks](https://mlsanity.com/paper/2310.19704/) (2023)
- [Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting](https://mlsanity.com/paper/2108.08165/) (2021)
- [Audio-Visual Class-Incremental Learning](https://mlsanity.com/paper/2308.11073/) (2023)
- [Class-Incremental Learning using Diffusion Model for Distillation and Replay](https://mlsanity.com/paper/2306.17560/) (2023)
- [Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer](https://mlsanity.com/paper/2101.09536/) (2021)
- [Balancing Stability and Plasticity through Advanced Null Space in Continual Learning](https://mlsanity.com/paper/2207.12061/) (2022)
- [Learnability and Algorithm for Continual Learning](https://mlsanity.com/paper/2306.12646/) (2023)
- [A Multi-Head Model for Continual Learning via Out-of-Distribution Replay](https://mlsanity.com/paper/2208.09734/) (2022)

## Abstract (arXiv)

Lifelong learning with deep neural networks is well-known to suffer from catastrophic forgetting: the performance on previous tasks drastically degrades when learning a new task. To alleviate this effect, we propose to leverage a large stream of unlabeled data easily obtainable in the wild. In particular, we design a novel class-incremental learning scheme with (a) a new distillation loss, termed global distillation, (b) a learning strategy to avoid overfitting to the most recent task, and (c) a confidence-based sampling method to effectively leverage unlabeled external data. Our experimental results on various datasets, including CIFAR and ImageNet, demonstrate the superiority of the proposed methods over prior methods, particularly when a stream of unlabeled data is accessible: our method shows up to 15.8% higher accuracy and 46.5% less forgetting compared to the state-of-the-art method. The code is available at https://github.com/kibok90/iccv2019-inc.