mlsanityData to 7 Oct 2026

Continual Learning of Diffusion Models with Generative Distillation

arXiv 2311.14028 ↗First posted 23 Nov 2023cs.LG cs.AI cs.CVcited by 19

On the map: Diffusion models → Robustness, privacy and adversarial aspects

Abstract (arXiv)

Diffusion models are powerful generative models that achieve state-of-the-art performance in image synthesis. However, training them demands substantial amounts of data and computational resources. Continual learning would allow for incrementally learning new tasks and accumulating knowledge, thus enabling the reuse of trained models for further learning. One potentially suitable continual learning approach is generative replay, where a copy of a generative model trained on previous tasks produces synthetic data that are interleaved with data from the current task. However, standard generative replay applied to diffusion models results in a catastrophic loss in denoising capabilities. In this paper, we propose generative distillation, an approach that distils the entire reverse process of a diffusion model. We demonstrate that our approach substantially improves the continual learning performance of generative replay with only a modest increase in the computational costs.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Robustness, privacy and adversarial aspects, one of 14 directions in Diffusion models. Builds on 32 of its 42 references that are in the map. First posted on arXiv 23 Nov 2023.

Builds on

32 of 42 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2016Learning without Forgettingcited by 2,465
  11. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
20 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  2. 2023Consistency Modelscited by 1,778
  3. 2016Progressive Neural Networkscited by 1,332
  4. 2017Continual Learning with Deep Generative Replaycited by 1,152
  5. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  6. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  7. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  8. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
  9. 2020Supermasks in Superpositioncited by 223
  10. 2020Continual Lifelong Learning in Natural Language Processing: A Surveycited by 152
  11. 2019Uncertainty-based Continual Learning with Adaptive Regularizationcited by 144
  12. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  13. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  14. 2023TRACT: Denoising Diffusion Models with Transitive Closure Time-Distillationcited by 115
  15. 2018Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilizationcited by 104
  16. 2023BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrappingcited by 96
  17. 2016Group Sparse Regularization for Deep Neural Networkscited by 89
  18. 2022On Analyzing Generative and Denoising Capabilities of Diffusion-based Deep Generative Modelscited by 31
  19. 2021Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environmentscited by 24
  20. 2023Exploring Continual Learning of Diffusion Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 8; 2026: 5.

19 papers.

Sort:

By area:

  1. 2024Schr"odinger Bridge Flow for Unpaired Data Translationcited by 47
  2. 2024ResAdapter: Domain Consistent Resolution Adapter for Diffusion Modelscited by 23
  3. 2024t-DGR: A Trajectory-Based Deep Generative Replay Method for Continual Learning in Decision Makingcited by 9
  4. 2026When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generatorscited by 6
  5. 2024GUIDE: Guidance-based Incremental Learning with Diffusion Modelscited by 5
  6. 2024CLoG: Benchmarking Continual Learning of Image Generation Modelscited by 5
  7. 2025Locality-Aware Continual Unlearning for Diffusion Modelscited by 5
  8. 2025Avoid Catastrophic Forgetting with Rank-1 Fisher from Diffusion Modelscited by 3
  9. 2024Lifelong Learning of Video Diffusion Models From a Single Video Streamcited by 2
  10. 2025PWD: Prior-Guided and Wavelet-Enhanced Diffusion Model for Limited-Angle CTcited by 1
  11. 2026Forget Less by Learning Together through Concept Consolidationcited by 1
  12. 2026Trust Region Continual Learning as an Implicit Meta-Learnercited by 1
7 more
  1. 2025CCD: Continual Consistency Diffusion for Lifelong Generative Modelingcited by 0
  2. 2025Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learningcited by 0
  3. 2025Beyond Discrete Samples: High Information Density Replay for Efficient Lifelong Person Re-Identificationcited by 0
  4. 2025VidCLearn: A Continual Learning Approach for Text-to-Video Generationcited by 0
  5. 2025Deep Generative Continual Learning using Functional LoRA: FunLoRAcited by 0
  6. 2026Continual Learning in Modern Hopfield Networks with an Application to Diffusion Modelscited by 0
  7. 2026Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Joint Diffusion models in Continual Learning
  2. 2023Class-Prototype Conditional Diffusion Model with Gradient Projection for Continual Learning
  3. 2022Progressive Distillation for Fast Sampling of Diffusion Models
  4. 2024Simple and Fast Distillation of Diffusion Models
  5. 2024Input-Adaptive Generative Dynamics in Diffusion Models
  6. 2025Generative Dataset Distillation using Min-Max Diffusion Model
  7. 2025A Survey on Pre-Trained Diffusion Model Distillations
  8. 2024DDIL: Diversity Enhancing Diffusion Distillation With Imitation Learning