# Self-distilled Knowledge Delegator for Exemplar-free Class Incremental Learning

arXiv: https://arxiv.org/abs/2205.11071 · HTML: https://mlsanity.com/paper/2205.11071/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 30 of 31)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Data-Free Quantization Through Weight Equalization and Bias Correction](https://mlsanity.com/paper/1906.04721/) (2019)
- [Semantic Drift Compensation for Class-Incremental Learning](https://mlsanity.com/paper/2004.00440/) (2020)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [FearNet: Brain-Inspired Model for Incremental Learning](https://mlsanity.com/paper/1711.10563/) (2017)
- [Zero-shot Knowledge Transfer via Adversarial Belief Matching](https://mlsanity.com/paper/1905.09768/) (2019)
- [Generative Feature Replay For Class-Incremental Learning](https://mlsanity.com/paper/2004.09199/) (2020)
- [Data-Free Adversarial Distillation](https://mlsanity.com/paper/1912.11006/) (2019)
- [Data-Free Network Quantization With Adversarial Knowledge Distillation](https://mlsanity.com/paper/2005.04136/) (2020)
- [Memory Replay GANs: learning to generate images from new categories without forgetting](https://mlsanity.com/paper/1809.02058/) (2018)
- [Generative Models from the perspective of Continual Learning](https://mlsanity.com/paper/1812.09111/) (2018)
- [The Knowledge Within: Methods for Data-Free Model Compression](https://mlsanity.com/paper/1912.01274/) (2019)
- [Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN](https://mlsanity.com/paper/2003.09088/) (2020)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier](https://mlsanity.com/paper/1912.11960/) (2019)
- [Ternary Feature Masks: zero-forgetting for task-incremental learning](https://mlsanity.com/paper/2001.08714/) (2020)

## Abstract (arXiv)

Exemplar-free incremental learning is extremely challenging due to inaccessibility of data from old tasks. In this paper, we attempt to exploit the knowledge encoded in a previously trained classification model to handle the catastrophic forgetting problem in continual learning. Specifically, we introduce a so-called knowledge delegator, which is capable of transferring knowledge from the trained model to a randomly re-initialized new model by generating informative samples. Given the previous model only, the delegator is effectively learned using a self-distillation mechanism in a data-free manner. The knowledge extracted by the delegator is then utilized to maintain the performance of the model on old tasks in incremental learning. This simple incremental learning framework surpasses existing exemplar-free methods by a large margin on four widely used class incremental benchmarks, namely CIFAR-100, ImageNet-Subset, Caltech-101 and Flowers-102. Notably, we achieve comparable performance to some exemplar-based methods without accessing any exemplars.