# Exploring Continual Learning of Diffusion Models

arXiv: https://arxiv.org/abs/2303.15342 · HTML: https://mlsanity.com/paper/2303.15342/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Continual Unsupervised Representation Learning](https://mlsanity.com/paper/1910.14481/) (2019)
- [Lifelong GAN: Continual Learning for Conditional Image Generation](https://mlsanity.com/paper/1907.10107/) (2019)
- [Memory Replay GANs: learning to generate images from new categories without forgetting](https://mlsanity.com/paper/1809.02058/) (2018)
- [Rehearsal revealed: The limits and merits of revisiting samples in continual learning](https://mlsanity.com/paper/2104.07446/) (2021)
- [Generative Models from the perspective of Continual Learning](https://mlsanity.com/paper/1812.09111/) (2018)
- [Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies](https://mlsanity.com/paper/1808.06508/) (2018)
- [On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning](https://mlsanity.com/paper/2210.06443/) (2022)
- [Learning latent representations across multiple data domains using Lifelong VAEGAN](https://mlsanity.com/paper/2007.10221/) (2020)
- [On Analyzing Generative and Denoising Capabilities of Diffusion-based Deep Generative Models](https://mlsanity.com/paper/2206.00070/) (2022)
- [BooVAE: Boosting Approach for Continual Learning of VAE](https://mlsanity.com/paper/1908.11853/) (2019)
- [Multiband VAE: Latent Space Alignment for Knowledge Consolidation in Continual Learning](https://mlsanity.com/paper/2106.12196/) (2021)

## Built on by (most cited first, 15 of 15)

- [Schr"odinger Bridge Flow for Unpaired Data Translation](https://mlsanity.com/paper/2409.09347/) (2024)
- [Continual Learning of Diffusion Models with Generative Distillation](https://mlsanity.com/paper/2311.14028/) (2023)
- [t-DGR: A Trajectory-Based Deep Generative Replay Method for Continual Learning in Decision Making](https://mlsanity.com/paper/2401.02576/) (2024)
- [Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replay](https://mlsanity.com/paper/2411.10809/) (2024)
- [Joint Diffusion models in Continual Learning](https://mlsanity.com/paper/2411.08224/) (2024)
- [GUIDE: Guidance-based Incremental Learning with Diffusion Models](https://mlsanity.com/paper/2403.03938/) (2024)
- [CLoG: Benchmarking Continual Learning of Image Generation Models](https://mlsanity.com/paper/2406.04584/) (2024)
- [CollaFuse: Collaborative Diffusion Models](https://mlsanity.com/paper/2406.14429/) (2024)
- [Low-Rank Continual Personalization of Diffusion Models](https://mlsanity.com/paper/2410.04891/) (2024)
- [Lifelong Learning of Video Diffusion Models From a Single Video Stream](https://mlsanity.com/paper/2406.04814/) (2024)
- [Forget Less by Learning Together through Concept Consolidation](https://mlsanity.com/paper/2601.01963/) (2026)
- [DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation](https://mlsanity.com/paper/2607.00183/) (2026)
- [Adapt & Align: Continual Learning with Generative Models Latent Space Alignment](https://mlsanity.com/paper/2312.13699/) (2023)
- [CCD: Continual Consistency Diffusion for Lifelong Generative Modeling](https://mlsanity.com/paper/2505.11936/) (2025)
- [Continual Learning in Modern Hopfield Networks with an Application to Diffusion Models](https://mlsanity.com/paper/2605.27975/) (2026)

## Abstract (arXiv)

Diffusion models have achieved remarkable success in generating high-quality images thanks to their novel training procedures applied to unprecedented amounts of data. However, training a diffusion model from scratch is computationally expensive. This highlights the need to investigate the possibility of training these models iteratively, reusing computation while the data distribution changes. In this study, we take the first step in this direction and evaluate the continual learning (CL) properties of diffusion models. We begin by benchmarking the most common CL methods applied to Denoising Diffusion Probabilistic Models (DDPMs), where we note the strong performance of the experience replay with the reduced rehearsal coefficient. Furthermore, we provide insights into the dynamics of forgetting, which exhibit diverse behavior across diffusion timesteps. We also uncover certain pitfalls of using the bits-per-dimension metric for evaluating CL.