mlsanityData to 14 Oct 2026

Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA

arXiv 2304.06027 ↗First posted 12 Apr 2023cs.CV cs.AI cs.LGcited by 141

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent works demonstrate a remarkable ability to customize text-to-image diffusion models while only providing a few example images. What happens if you try to customize such models using multiple, fine-grained concepts in a sequential (i.e., continual) manner? In our work, we show that recent state-of-the-art customization of text-to-image models suffer from catastrophic forgetting when new concepts arrive sequentially. Specifically, when adding a new concept, the ability to generate high quality images of past, similar concepts degrade. To circumvent this forgetting, we propose a new method, C-LoRA, composed of a continually self-regularized low-rank adaptation in cross attention layers of the popular Stable Diffusion model. Furthermore, we use customization prompts which do not include the word of the customized object (i.e., "person" for a human face dataset) and are initialized as completely random embeddings. Importantly, our method induces only marginal additional parameter costs and requires no storage of user data for replay. We show that C-LoRA not only outperforms several baselines for our proposed setting of text-to-image continual customization, which we refer to as Continual Diffusion, but that we achieve a new state-of-the-art in the well-established rehearsal-free continual learning setting for image classification. The high achieving performance of C-LoRA in two separate domains positions it as a compelling solution for a wide range of applications, and we believe it has significant potential for practical impact. Project page: https://jamessealesmith.github.io/continual-diffusion/

Key facts. Cited by 141 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 58 of its 75 references that are in the map. First posted on arXiv 12 Apr 2023.

Builds on

58 of 75 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  3. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  6. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  7. 2016Learning without Forgettingcited by 2,496
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  11. 2014Conditional Generative Adversarial Netscited by 2,162
  12. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  13. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  14. 2016Progressive Neural Networkscited by 1,361
  15. 2018Demystifying MMD GANscited by 1,293
  16. 2017Continual Learning with Deep Generative Replaycited by 1,158
  17. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
  18. 2018Experience Replay for Continual Learningcited by 1,095
  19. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
  20. 2021Learning to Prompt for Continual Learningcited by 944
  21. 2017Lifelong Learning with Dynamically Expandable Networkscited by 694
  22. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  23. 2019Three scenarios for continual learningcited by 614
  24. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 450
  25. 2016Expert Gate: Lifelong Learning with a Network of Expertscited by 372
  26. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  27. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 309
  28. 2019On Tiny Episodic Memories in Continual Learningcited by 307
  29. 2019Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgettingcited by 302
  30. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  31. 2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
  32. 2019Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learningcited by 182
  33. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  34. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  35. 2021Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learningcited by 131
  36. 2022DigiFace-1M: 1 Million Digital Face Images for Face Recognitioncited by 92
  37. 2019Lifelong GAN: Continual Learning for Conditional Image Generationcited by 78
  38. 2018Generative Models from the perspective of Continual Learningcited by 69

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 23; 2024: 62; 2025: 36; 2026: 20.

141 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  3. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
  4. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  5. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  6. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 173
  7. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 139
  8. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
  9. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  10. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 93
  11. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
  12. 2023PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalizationcited by 77
38 more
  1. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  2. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 66
  3. 2024OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Modelscited by 64
  4. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  5. 2023Inserting Anybody in Diffusion Models via Celeb Basiscited by 63
  6. 2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Modelscited by 55
  7. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  8. 2023Continual Instruction Tuning for Large Multimodal Modelscited by 49
  9. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  10. 2023Customizing 360-Degree Panoramas through Text-to-Image Diffusion Modelscited by 47
  11. 2024Schr"odinger Bridge Flow for Unpaired Data Translationcited by 47
  12. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  13. 2024Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditionscited by 42
  14. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 41
  15. 2024How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?cited by 33
  16. 2024IDAdapter: Learning Mixed Features for Tuning-Free Personalization of Text-to-Image Modelscited by 32
  17. 2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 32
  18. 2024HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Groundingcited by 31
  19. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  20. 2025CL-LoRA: Continual Low-Rank Adaptation for Rehearsal-Free Class-Incremental Learningcited by 28
  21. 2024BootPIG: Bootstrapping Zero-shot Personalized Image Generation Capabilities in Pretrained Diffusion Modelscited by 27
  22. 2024Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptationcited by 27
  23. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
  24. 2024ResAdapter: Domain Consistent Resolution Adapter for Diffusion Modelscited by 23
  25. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  26. 2024Group Diffusion Transformers are Unsupervised Multitask Learnerscited by 22
  27. 2023Continual Learning of Diffusion Models with Generative Distillationcited by 20
  28. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuningcited by 19
  29. 2023HiPA: Enabling One-Step Text-to-Image Diffusion Models via High-Frequency-Promoting Adaptationcited by 16
  30. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  31. 2025T-LoRA: Single Image Diffusion Model Customization Without Overfittingcited by 15
  32. 2025Skill Expansion and Composition in Parameter Spacecited by 14
  33. 2025Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Modelscited by 13
  34. 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuningcited by 13
  35. 2024LLMs Can Evolve Continually on Modality for X-Modal Reasoningcited by 12
  36. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 12
  37. 2024t-DGR: A Trajectory-Based Deep Generative Replay Method for Continual Learning in Decision Makingcited by 11
  38. 2025Continual Gradient Low-Rank Projection Fine-Tuning for LLMscited by 11

50 most cited of 141 All 141 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models
  2. 2025Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift
  3. 2026LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models
  4. 2025TARA: Token-Aware LoRA for Composable Personalization in Diffusion Models
  5. 2026SeqLoRA: Bilevel Orthogonal Adaptation for Continual Multi-Concept Generation
  6. 2024Direct Consistency Optimization for Robust Customization of Text-to-Image Diffusion Models
  7. 2025Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective
  8. 2024LoRA Diffusion: Zero-Shot LoRA Synthesis for Diffusion Model Personalization