# Class-Incremental Learning using Diffusion Model for Distillation and Replay

arXiv: https://arxiv.org/abs/2306.17560 · HTML: https://mlsanity.com/paper/2306.17560/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models.
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 30 Jun 2023.

## Placed in

- Diffusion models / [Robustness, privacy and adversarial aspects](https://mlsanity.com/diffusion/robustness-privacy-and-adversarial-aspects/)

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [DER: Dynamically Expandable Representation for Class Incremental Learning](https://mlsanity.com/paper/2103.16788/) (2021)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [FOSTER: Feature Boosting and Compression for Class-Incremental Learning](https://mlsanity.com/paper/2204.04662/) (2022)
- [Mnemonics Training: Multi-Class Incremental Learning without Forgetting](https://mlsanity.com/paper/2002.10211/) (2020)
- [Class-Incremental Learning: A Survey](https://mlsanity.com/paper/2302.03648/) (2023)
- [Class-incremental Learning via Deep Model Consolidation](https://mlsanity.com/paper/1903.07864/) (2019)
- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning](https://mlsanity.com/paper/2106.09701/) (2021)
- [SS-IL: Separated Softmax for Incremental Learning](https://mlsanity.com/paper/2003.13947/) (2020)
- [Distilling Causal Effect of Data in Class-Incremental Learning](https://mlsanity.com/paper/2103.01737/) (2021)
- [Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild](https://mlsanity.com/paper/1903.12648/) (2019)
- [Generative Models as a Data Source for Multiview Representation Learning](https://mlsanity.com/paper/2106.05258/) (2021)
- [ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning](https://mlsanity.com/paper/2101.00407/) (2021)
- [This dataset does not exist: training models from generated images](https://mlsanity.com/paper/1911.02888/) (2019)
- [Balanced softmax cross-entropy for incremental learning with and without memory](https://mlsanity.com/paper/2103.12532/) (2021)

## Built on by (most cited first, 30 of 31)

- [Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters](https://mlsanity.com/paper/2403.11549/) (2024)
- [A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning](https://mlsanity.com/paper/2307.09218/) (2023)
- [SDDGR: Stable Diffusion-based Deep Generative Replay for Class Incremental Object Detection](https://mlsanity.com/paper/2402.17323/) (2024)
- [DiffClass: Diffusion-Based Class Incremental Learning](https://mlsanity.com/paper/2403.05016/) (2024)
- [VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model](https://mlsanity.com/paper/2403.05346/) (2024)
- [Synthetic Data is an Elegant GIFT for Continual Vision-Language Models](https://mlsanity.com/paper/2503.04229/) (2025)
- [Never-Ending Behavior-Cloning Agent for Robotic Manipulation](https://mlsanity.com/paper/2403.00336/) (2024)
- [GUIDE: Guidance-based Incremental Learning with Diffusion Models](https://mlsanity.com/paper/2403.03938/) (2024)
- [Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learning](https://mlsanity.com/paper/2403.07356/) (2024)
- [Future-Proofing Class-Incremental Learning](https://mlsanity.com/paper/2404.03200/) (2024)
- [Diffusion Model Meets Non-Exemplar Class-Incremental Learning and Beyond](https://mlsanity.com/paper/2408.02983/) (2024)
- [Low-Rank Continual Personalization of Diffusion Models](https://mlsanity.com/paper/2410.04891/) (2024)
- [Divide-and-Conquer for Enhancing Unlabeled Learning, Stability, and Plasticity in Semi-supervised Continual Learning](https://mlsanity.com/paper/2508.05316/) (2025)
- [Prompt-Based Exemplar Super-Compression and Regeneration for Class-Incremental Learning](https://mlsanity.com/paper/2311.18266/) (2023)
- [Replay to Remember (R2R): An Efficient Uncertainty-driven Unsupervised Continual Learning Framework Using Generative Replay](https://mlsanity.com/paper/2505.04787/) (2025)
- [LoRA-Loop: Closing the Synthetic Replay Cycle for Continual VLM Learning](https://mlsanity.com/paper/2507.13568/) (2025)
- [Can Synthetic Images Conquer Forgetting? Beyond Unexplored Doubts in Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2507.13739/) (2025)
- [Exploiting Fine-Grained Prototype Distribution for Boosting Unsupervised Class Incremental Learning](https://mlsanity.com/paper/2408.10046/) (2024)
- [Beyond Synthetic Replays: Turning Diffusion Features into Few-Shot Class-Incremental Learning Knowledge](https://mlsanity.com/paper/2503.23402/) (2025)
- [Recommendation of data-free class-incremental learning algorithms by simulating future data](https://mlsanity.com/paper/2403.18132/) (2024)
- [Caption, Create, Continue: Continual Learning with Pre-trained Generative Vision-Language Models](https://mlsanity.com/paper/2409.17806/) (2024)
- [Efficient Non-Exemplar Class-Incremental Learning with Retrospective Feature Synthesis](https://mlsanity.com/paper/2411.01465/) (2024)
- [Fresh-CL: Feature Realignment through Experts on Hypersphere in Continual Learning](https://mlsanity.com/paper/2501.02198/) (2025)
- [KDC-Diff: A Latent-Aware Diffusion Model with Knowledge Retention for Memory-Efficient Image Generation](https://mlsanity.com/paper/2505.06995/) (2025)
- [Deep Generative Continual Learning using Functional LoRA: FunLoRA](https://mlsanity.com/paper/2510.02631/) (2025)
- [Dream2Learn: Structured Generative Dreaming for Continual Learning](https://mlsanity.com/paper/2603.01935/) (2026)
- [Temporal Memory for Resource-Constrained Agents: Continual Learning via Stochastic Compress-Add-Smooth](https://mlsanity.com/paper/2604.00067/) (2026)
- [SR2-LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning](https://mlsanity.com/paper/2605.07420/) (2026)
- [Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning](https://mlsanity.com/paper/2605.14785/) (2026)
- [DRDN: Decoupled Representation Dynamic Network for From-Scratch ViT Class-Incremental Learning](https://mlsanity.com/paper/2607.01630/) (2026)

## Abstract (arXiv)

Class-incremental learning aims to learn new classes in an incremental fashion without forgetting the previously learned ones. Several research works have shown how additional data can be used by incremental models to help mitigate catastrophic forgetting. In this work, following the recent breakthrough in text-to-image generative models and their wide distribution, we propose the use of a pretrained Stable Diffusion model as a source of additional data for class-incremental learning. Compared to competitive methods that rely on external, often unlabeled, datasets of real images, our approach can generate synthetic samples belonging to the same classes as the previously encountered images. This allows us to use those additional data samples not only in the distillation loss but also for replay in the classification loss. Experiments on the competitive benchmarks CIFAR100, ImageNet-Subset, and ImageNet demonstrate how this new approach can be used to further improve the performance of state-of-the-art methods for class-incremental learning on large scale datasets.