mlsanityData to 14 Oct 2026

Modular Customization of Diffusion Models via Blockwise-Parameterized Low-Rank Adaptation

arXiv 2503.08575 ↗First posted 11 Mar 2025cs.CVcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent diffusion model customization has shown impressive results in incorporating subject or style concepts with a handful of images. However, the modular composition of multiple concepts into a customized model, aimed to efficiently merge decentralized-trained concepts without influencing their identities, remains unresolved. Modular customization is essential for applications like concept stylization and multi-concept customization using concepts trained by different users. Existing post-training methods are only confined to a fixed set of concepts, and any different combinations require a new round of retraining. In contrast, instant merging methods often cause identity loss and interference of individual merged concepts and are usually limited to a small number of concepts. To address these issues, we propose BlockLoRA, an instant merging method designed to efficiently combine multiple concepts while accurately preserving individual concepts' identity. With a careful analysis of the underlying reason for interference, we develop the Randomized Output Erasure technique to minimize the interference of different customized models. Additionally, Blockwise LoRA Parameterization is proposed to reduce the identity loss during instant model merging. Extensive experiments validate the effectiveness of BlockLoRA, which can instantly merge 15 concepts of people, subjects, scenes, and styles with high fidelity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 35 of its 38 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

35 of 38 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Orthogonal Adaptation for Modular Customization of Diffusion Models5 Dec
  2. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models29 May
  3. 2023Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models5 Apr
  4. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuning20 Mar
  5. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
14 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  2. 2022Editing Models with Task Arithmeticcited by 1,120
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  4. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  5. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  6. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 416
  7. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  8. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  9. 2018DropBlock: A regularization method for convolutional networkscited by 262
  10. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
  11. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 225
  12. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  13. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  14. 2024Mixture of LoRA Expertscited by 181

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025T-LoRA: Single Image Diffusion Model Customization Without Overfitting
  2. 2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models
  3. 2026LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models
  4. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA
  5. 2024Low-Rank Continual Personalization of Diffusion Models
  6. 2024LoRACLR: Contrastive Adaptation for Customization of Diffusion Models
  7. 2025Parallel Rescaling: Rebalancing Consistency Guidance for Personalized Diffusion Models
  8. 2025TARA: Token-Aware LoRA for Composable Personalization in Diffusion Models