# Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models

arXiv: https://arxiv.org/abs/2404.03913 · HTML: https://mlsanity.com/paper/2404.03913/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 30 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [Inversion-Based Style Transfer with Diffusion Models](https://mlsanity.com/paper/2211.13203/) (2022)
- [Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models](https://mlsanity.com/paper/2305.18292/) (2023)
- [UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild](https://mlsanity.com/paper/2305.11147/) (2023)
- [Key-Locked Rank One Editing for Text-to-Image Personalization](https://mlsanity.com/paper/2305.01644/) (2023)
- [Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning](https://mlsanity.com/paper/2309.02591/) (2023)
- [Improving Tuning-Free Real Image Editing with Proximal Guidance](https://mlsanity.com/paper/2306.05414/) (2023)
- [Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.09869/) (2023)
- [Generate Anything Anywhere in Any Scene](https://mlsanity.com/paper/2306.17154/) (2023)

## Built on by (most cited first, 23 of 23)

- [Less-to-More Generalization: Unlocking More Controllability by In-Context Generation](https://mlsanity.com/paper/2504.02160/) (2025)
- [UniPortrait: A Unified Framework for Identity-Preserving Single- and Multi-Human Image Personalization](https://mlsanity.com/paper/2408.05939/) (2024)
- [Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Prompts](https://mlsanity.com/paper/2502.07802/) (2025)
- [Personalize Anything for Free with Diffusion Transformer](https://mlsanity.com/paper/2503.12590/) (2025)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance](https://mlsanity.com/paper/2405.17532/) (2024)
- [AnyStory: Towards Unified Single and Multiple Subject Personalization in Text-to-Image Generation](https://mlsanity.com/paper/2501.09503/) (2025)
- [TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation](https://mlsanity.com/paper/2410.05591/) (2024)
- [LatexBlend: Scaling Multi-concept Customized Generation with Latent Textual Blending](https://mlsanity.com/paper/2503.06956/) (2025)
- [MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidance](https://mlsanity.com/paper/2505.20525/) (2025)
- [Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis](https://mlsanity.com/paper/2408.03632/) (2024)
- [Continual Personalization for Diffusion Models](https://mlsanity.com/paper/2510.02296/) (2025)
- [FlipConcept: Tuning-Free Multi-Concept Personalization for Text-to-Image Generation](https://mlsanity.com/paper/2502.15203/) (2025)
- [Composing Concepts from Images and Videos via Concept-prompt Binding](https://mlsanity.com/paper/2512.09824/) (2025)
- [SPDiffusion: Semantic Protection Diffusion Models for Multi-concept Text-to-image Generation](https://mlsanity.com/paper/2409.01327/) (2024)
- [JEDI: The Force of Jensen-Shannon Divergence in Disentangling Diffusion Models](https://mlsanity.com/paper/2505.19166/) (2025)
- [AnyMS: Bottom-up Attention Decoupling for Layout-guided and Training-free Multi-subject Customization](https://mlsanity.com/paper/2512.23537/) (2025)
- [FaR: Enhancing Multi-Concept Text-to-Image Diffusion via Concept Fusion and Localized Refinement](https://mlsanity.com/paper/2504.03292/) (2025)
- [FreeFuse: Multi-Subject LoRA Fusion via Adaptive Token-Level Routing at Test Time](https://mlsanity.com/paper/2510.23515/) (2025)
- [A Training-Free Approach for Multi-ID Customization via Attention Adjustment and Spatial Control](https://mlsanity.com/paper/2511.20401/) (2025)
- [Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting](https://mlsanity.com/paper/2606.03792/) (2026)
- [MultiAnimate: A Unified Framework for Controllable Multi-Character Animation](https://mlsanity.com/paper/2607.13415/) (2026)
- [X-MULTI: VLM-based Imaging Factor Disentanglement for Factor-Aware Image Synthesis](https://mlsanity.com/paper/2608.24563/) (2026)

## Abstract (arXiv)

While there has been significant progress in customizing text-to-image generation models, generating images that combine multiple personalized concepts remains challenging. In this work, we introduce Concept Weaver, a method for composing customized text-to-image diffusion models at inference time. Specifically, the method breaks the process into two steps: creating a template image aligned with the semantics of input prompts, and then personalizing the template using a concept fusion strategy. The fusion strategy incorporates the appearance of the target concepts into the template image while retaining its structural details. The results indicate that our method can generate multiple custom concepts with higher identity fidelity compared to alternative approaches. Furthermore, the method is shown to seamlessly handle more than two concepts and closely follow the semantic meaning of the input prompt without blending appearances across different subjects.