MUSE: Multi-Subject Unified Synthesis via Explicit Layout Semantic Expansion
On the map: Diffusion models → Image generation, text-to-image and editing
Abstract (arXiv)
Existing text-to-image diffusion models have demonstrated remarkable capabilities in generating high-quality images guided by textual prompts. However, achieving multi-subject compositional synthesis with precise spatial control remains a significant challenge. In this work, we address the task of layout-controllable multi-subject synthesis (LMS), which requires both faithful reconstruction of reference subjects and their accurate placement in specified regions within a unified image. While recent advancements have separately improved layout control and subject synthesis, existing approaches struggle to simultaneously satisfy the dual requirements of spatial precision and identity preservation in this composite task. To bridge this gap, we propose MUSE, a unified synthesis framework that employs concatenated cross-attention (CCA) to seamlessly integrate layout specifications with textual guidance through explicit semantic space expansion. The proposed CCA mechanism enables bidirectional modality alignment between spatial constraints and textual descriptions without interference. Furthermore, we design a progressive two-stage training strategy that decomposes the LMS task into learnable sub-objectives for effective optimization. Extensive experiments demonstrate that MUSE achieves zero-shot end-to-end generation with superior spatial accuracy and identity consistency compared to existing solutions, advancing the frontier of controllable image synthesis. Our code and model are available at https://github.com/pf0607/MUSE.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 61 of its 68 references that are in the map. First posted on arXiv 20 Aug 2025.
Builds on
61 of 68 references.
Closest ancestors · 26
Same direction, newest first
- 2024IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation12 Sep
- 2024Move Anything with Layered Scene Diffusion10 Apr
- 2024NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging6 Mar
- 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space7 Feb
- 2024InstanceDiffusion: Instance-level Control for Image Generation5 Feb
- 2024InstantID: Zero-shot Identity-Preserving Generation in Seconds15 Jan
- 2023Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion5 Oct
- 2023Dense Text-to-Image Generation with Attention Modulation24 Aug
- 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
- 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning21 Jul
- 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
- 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuning12 Jun
14 more
- 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models29 May
- 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuning20 Mar
- 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
- 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
- 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
- 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
- 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
- 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
- 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
- 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
- 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
Other references · 35
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2020Denoising Diffusion Implicit Modelscited by 9,781
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
- 2022Classifier-Free Diffusion Guidancecited by 5,847
- 2022Generative Adversarial Networkscited by 5,627
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
23 more
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
- 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
- 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
- 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
- 2016Generative Adversarial Text to Image Synthesiscited by 735
- 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
- 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
- 2023Generative Multimodal Models are In-Context Learnerscited by 446
- 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
- 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 416
- 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 412
- 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
- 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
- 2023Generating Images with Multimodal Language Modelscited by 260
- 2022ReCo: Region-Controlled Text-to-Image Generationcited by 179
- 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 144
- 2023SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generationcited by 143
- 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 124
- 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
- 2024RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customizationcited by 39
- 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35
Built on by
Citing papers per year: 2025: 2; 2026: 2.
4 papers.
- 2025ConsistCompose: Unified Multimodal Layout Control for Image Compositioncited by 5
- 2025Borrowing from anything: A generalizable framework for reference-guided instance editingcited by 0
- 2026RAVA: Retrieval-Augmented Viewpoint Alignment for Subject-Driven Image Generationcited by 0
- 2026OccluRank: Controllable Occlusion-Aware Layout-to-Image Generation by Adding Just an Ordinal Rankcited by 0
- 2026OccluRank: Controllable Occlusion-Aware Layout-to-Image Generation by Adding Just an Ordinal Rank21 Aug 2026
- 2026RAVA: Retrieval-Augmented Viewpoint Alignment for Subject-Driven Image Generation16 Jun 2026
- 2025Borrowing from anything: A generalizable framework for reference-guided instance editing17 Dec 2025
- 2025ConsistCompose: Unified Multimodal Layout Control for Image Composition23 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2024Obtaining Favorable Layouts for Multiple Object Generation
- 2025HCMA: Hierarchical Cross-model Alignment for Grounded Text-to-Image Generation
- 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation
- 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis
- 2024Training-free Composite Scene Generation for Layout-to-Image Synthesis
- 2025A Two-Stage System for Layout-Controlled Image Generation using Large Language Models and Diffusion Models
- 2026All-in-One Conditioning for Text-to-Image Synthesis
- 2024Layout-to-Image Generation with Localized Descriptions using ControlNet with Cross-Attention Control