mlsanityData to 7 Oct 2026

Multi-Concept Customization of Text-to-Image Diffusion

arXiv 2212.04488 ↗First posted 8 Dec 2022cs.CV cs.GR cs.LGcited by 1,099

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

While generative models produce high-quality images of concepts learned from a large-scale database, a user often wishes to synthesize instantiations of their own concepts (for example, their family, pets, or items). Can we teach a model to quickly acquire a new concept, given a few examples? Furthermore, can we compose multiple new concepts together? We propose Custom Diffusion, an efficient method for augmenting existing text-to-image models. We find that only optimizing a few parameters in the text-to-image conditioning mechanism is sufficiently powerful to represent new concepts while enabling fast tuning (~6 minutes). Additionally, we can jointly train for multiple concepts or combine multiple fine-tuned models into one via closed-form constrained optimization. Our fine-tuned model generates variations of multiple new concepts and seamlessly composes them with existing concepts in novel settings. Our method outperforms or performs on par with several baselines and concurrent works in both qualitative and quantitative evaluations while being memory and computationally efficient.

Key facts. Cited by 1,099 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 80 of its 96 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

80 of 96 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022On the detection of synthetic images generated by diffusion models1 Nov
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  3. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  7. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  12. 2022Generative Adversarial Networkscited by 5,603
38 more
  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  7. 2021Convex optimizationcited by 2,479
  8. 2016Learning without Forgettingcited by 2,465
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  13. 2016Density estimation using Real NVPcited by 2,029
  14. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  15. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  16. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  17. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  18. 2018Demystifying MMD GANscited by 1,268
  19. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  20. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  21. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  22. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  23. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  24. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  25. 2019CNN-generated images are surprisingly easy to spot... for nowcited by 842
  26. 2016Pixel Recurrent Neural Networkscited by 814
  27. 2016Generative Adversarial Text to Image Synthesiscited by 735
  28. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  29. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  30. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  31. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  32. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  33. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  34. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  35. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  36. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  37. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  38. 2020What makes fake images detectable? Understanding properties that generalizecited by 272

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 229; 2024: 370; 2025: 348; 2026: 151.

1,099 papers.

Sort:

By area:

  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  2. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  3. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2023Erasing Concepts from Diffusion Modelscited by 530
  6. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  7. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  8. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  9. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  10. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 411
  11. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  12. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
38 more
  1. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 334
  2. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  3. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  4. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  5. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  6. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  7. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  8. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  9. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  10. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  11. 2023Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylizationcited by 251
  12. 2023Personalize Segment Anything Model with One Shotcited by 250
  13. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  14. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  15. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  16. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  17. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
  18. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  19. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  20. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  21. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  22. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  23. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  24. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  25. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  26. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  27. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  28. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  29. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  30. 2024Mixture of LoRA Expertscited by 179
  31. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  32. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  33. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
  34. 2023A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpaintingcited by 168
  35. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  36. 2023UFOGen: You Forward Once Large Scale Text-to-Image Generation via Diffusion GANscited by 165
  37. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  38. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155

50 most cited of 1,099 All 1,099 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Generating images of rare concepts using pre-trained diffusion models
  2. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  3. 2024LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation
  4. 2025Blending Concepts with Text-to-Image Diffusion Models
  5. 2024Towards Lifelong Few-Shot Customization of Text-to-Image Diffusion
  6. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  7. 2023Text-to-image Diffusion Models in Generative AI: A Survey
  8. 2026Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation