# A Strategy for an Uncompromising Incremental Learner

arXiv: https://arxiv.org/abs/1705.00744 · HTML: https://mlsanity.com/paper/1705.00744/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2017.

## Built on by (most cited first, 9 of 9)

- [FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2211.13131/) (2022)
- [Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks](https://mlsanity.com/paper/2112.10017/) (2021)
- [Learning with Recoverable Forgetting](https://mlsanity.com/paper/2207.08224/) (2022)
- [Unsupervised Continual Learning Via Pseudo Labels](https://mlsanity.com/paper/2104.07164/) (2021)
- [Online Continual Learning Via Candidates Voting](https://mlsanity.com/paper/2110.08855/) (2021)
- [Cold Start Streaming Learning for Deep Networks](https://mlsanity.com/paper/2211.04624/) (2022)
- [PointCLIMB: An Exemplar-Free Point Cloud Class Incremental Benchmark](https://mlsanity.com/paper/2304.06775/) (2023)
- [A Comparative Study of Calibration Methods for Imbalanced Class Incremental Learning](https://mlsanity.com/paper/2202.00386/) (2022)
- [DILF-EN framework for Class-Incremental Learning](https://mlsanity.com/paper/2112.12385/) (2021)

## Abstract (arXiv)

Multi-class supervised learning systems require the knowledge of the entire range of labels they predict. Often when learnt incrementally, they suffer from catastrophic forgetting. To avoid this, generous leeways have to be made to the philosophy of incremental learning that either forces a part of the machine to not learn, or to retrain the machine again with a selection of the historic data. While these hacks work to various degrees, they do not adhere to the spirit of incremental learning. In this article, we redefine incremental learning with stringent conditions that do not allow for any undesirable relaxations and assumptions. We design a strategy involving generative models and the distillation of dark knowledge as a means of hallucinating data along with appropriate targets from past distributions. We call this technique, phantom sampling.We show that phantom sampling helps avoid catastrophic forgetting during incremental learning. Using an implementation based on deep neural networks, we demonstrate that phantom sampling dramatically avoids catastrophic forgetting. We apply these strategies to competitive multi-class incremental learning of deep neural networks. Using various benchmark datasets and through our strategy, we demonstrate that strict incremental learning could be achieved. We further put our strategy to test on challenging cases, including cross-domain increments and incrementing on a novel label space. We also propose a trivial extension to unbounded-continual learning and identify potential for future development.