# InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention

arXiv: https://arxiv.org/abs/2509.16691 · HTML: https://mlsanity.com/paper/2509.16691/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 59 of its 65 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 59)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [All are Worth Words: A ViT Backbone for Diffusion Models](https://mlsanity.com/paper/2209.12152/) (2022)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding](https://mlsanity.com/paper/2405.08748/) (2024)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](https://mlsanity.com/paper/2303.17189/) (2023)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)
- [InstanceDiffusion: Instance-level Control for Image Generation](https://mlsanity.com/paper/2402.03290/) (2024)
- [Grounded Text-to-Image Synthesis with Attention Refocusing](https://mlsanity.com/paper/2306.05427/) (2023)

## Built on by (most cited first, 10 of 10)

- [FireRed-Image-Edit-1.0 Technical Report](https://mlsanity.com/paper/2602.13344/) (2026)
- [Layer-wise Instance Binding for Regional and Occlusion Control in Text-to-Image Diffusion Transformers](https://mlsanity.com/paper/2603.05769/) (2026)
- [TexTailor: Inference-Time Textual Guidance Tailoring for Multimodal Diffusion Transformers](https://mlsanity.com/paper/2601.02211/) (2026)
- [DepthArb: Training-Free Depth-Arbitrated Generation for Occlusion-Robust Image Synthesis](https://mlsanity.com/paper/2603.23924/) (2026)
- [OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generation](https://mlsanity.com/paper/2605.21343/) (2026)
- [COLLAR: Cascaded Object-Level Latent Refinement for High-Fidelity Conditional Generation](https://mlsanity.com/paper/2606.00954/) (2026)
- [Bridging Language and Spherical Space: Object-Centric Control for Text-to-Panorama Generation](https://mlsanity.com/paper/2608.20691/) (2026)
- [OccluRank: Controllable Occlusion-Aware Layout-to-Image Generation by Adding Just an Ordinal Rank](https://mlsanity.com/paper/2608.20932/) (2026)
- [RefCompose: Multi-Reference Image Generation via LoRA-Conditioned Diffusion](https://mlsanity.com/paper/2609.32389/) (2026)
- [OverLay++: Dense-Overlap Layout-to-Image Generation Dataset](https://mlsanity.com/paper/2610.09071/) (2026)

## Abstract (arXiv)

Diffusion models have demonstrated remarkable capabilities in generating high-quality images. Recent advancements in Layout-to-Image (L2I) generation have leveraged positional conditions and textual descriptions to facilitate precise and controllable image synthesis. Despite overall progress, current L2I methods still exhibit suboptimal performance. Therefore, we propose InstanceAssemble, a novel architecture that incorporates layout conditions via instance-assembling attention, enabling position control with bounding boxes (bbox) and multimodal content control including texts and additional visual content. Our method achieves flexible adaption to existing DiT-based T2I models through light-weighted LoRA modules. Additionally, we propose a Layout-to-Image benchmark, Denselayout, a comprehensive benchmark for layout-to-image generation, containing 5k images with 90k instances in total. We further introduce Layout Grounding Score (LGS), an interpretable evaluation metric to more precisely assess the accuracy of L2I generation. Experiments demonstrate that our InstanceAssemble method achieves state-of-the-art performance under complex layout conditions, while exhibiting strong compatibility with diverse style LoRA modules. The code and pretrained models are publicly available at https://github.com/FireRedTeam/InstanceAssemble.