# Compositional Text-to-Image Generation with Dense Blob Representations

arXiv: https://arxiv.org/abs/2405.08246 · HTML: https://mlsanity.com/paper/2405.08246/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 58 references that are in the map.
- First posted on arXiv 14 May 2024.

## Builds on (most cited first, 30 of 51)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)

## Built on by (most cited first, 30 of 38)

- [CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation](https://mlsanity.com/paper/2412.03859/) (2024)
- [GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Models](https://mlsanity.com/paper/2407.02252/) (2024)
- [DiffUHaul: A Training-Free Method for Object Dragging in Images](https://mlsanity.com/paper/2406.01594/) (2024)
- [Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation](https://mlsanity.com/paper/2501.03059/) (2025)
- [Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddings](https://mlsanity.com/paper/2406.07844/) (2024)
- [Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation](https://mlsanity.com/paper/2408.14819/) (2024)
- [PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models](https://mlsanity.com/paper/2503.10127/) (2025)
- [BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations](https://mlsanity.com/paper/2501.07647/) (2025)
- [Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling](https://mlsanity.com/paper/2405.21048/) (2024)
- [Spatial Transport Optimization by Repositioning Attention Map for Training-Free Text-to-Image Synthesis](https://mlsanity.com/paper/2503.22168/) (2025)
- [Be Decisive: Noise-Induced Layouts for Multi-Subject Generation](https://mlsanity.com/paper/2505.21488/) (2025)
- [Wavelets Are All You Need for Autoregressive Image Generation](https://mlsanity.com/paper/2406.19997/) (2024)
- [Relations, Negations, and Numbers: Looking for Logic in Generative Text-to-Image Models](https://mlsanity.com/paper/2411.17066/) (2024)
- [ROICtrl: Boosting Instance Control for Visual Generation](https://mlsanity.com/paper/2411.17949/) (2024)
- [Beyond Simple Edits: X-Planner for Complex Instruction-Based Image Editing](https://mlsanity.com/paper/2507.05259/) (2025)
- [SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation](https://mlsanity.com/paper/2511.16666/) (2025)
- [VSC: Visual Search Compositional Text-to-Image Diffusion Model](https://mlsanity.com/paper/2505.01104/) (2025)
- [LOTS of Fashion! Multi-Conditioning for Image Generation via Sketch-Text Pairing](https://mlsanity.com/paper/2507.22627/) (2025)
- [Controllable Human Image Generation with Personalized Multi-Garments](https://mlsanity.com/paper/2411.16801/) (2024)
- [GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis](https://mlsanity.com/paper/2412.06089/) (2024)
- [Inverse Virtual Try-On: Generating Multi-Category Product-Style Images from Clothed Individuals](https://mlsanity.com/paper/2505.21062/) (2025)
- [BlobCtrl: Taming Controllable Blob for Element-level Image Editing](https://mlsanity.com/paper/2503.13434/) (2025)
- [LLMControl: Grounded Control of Text-to-Image Diffusion-based Synthesis with Multimodal LLMs](https://mlsanity.com/paper/2507.19939/) (2025)
- [Envisioning Beyond the Few: Disentangled Semantics and Primitives for Few-Shot Atypical Layout-to-Image Generation](https://mlsanity.com/paper/2605.31266/) (2026)
- [Generating Intermediate Representations for Compositional Text-To-Image Generation](https://mlsanity.com/paper/2410.09792/) (2024)
- [CardioComposer: Leveraging Differentiable Geometry for Compositional Control of Anatomical Diffusion Models](https://mlsanity.com/paper/2509.08015/) (2025)
- [InfSplign: Inference-Time Spatial Alignment of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2512.17851/) (2025)
- [DrivePTS: A Progressive Learning Framework with Textual and Structural Enhancement for Driving Scene Generation](https://mlsanity.com/paper/2602.22549/) (2026)
- [From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing](https://mlsanity.com/paper/2605.15181/) (2026)
- [Improving Fine-Grained Control via Aggregation of Multiple Diffusion Models](https://mlsanity.com/paper/2410.01262/) (2024)

## Abstract (arXiv)

Existing text-to-image models struggle to follow complex text prompts, raising the need for extra grounding inputs for better controllability. In this work, we propose to decompose a scene into visual primitives - denoted as dense blob representations - that contain fine-grained details of the scene while being modular, human-interpretable, and easy-to-construct. Based on blob representations, we develop a blob-grounded text-to-image diffusion model, termed BlobGEN, for compositional generation. Particularly, we introduce a new masked cross-attention module to disentangle the fusion between blob representations and visual features. To leverage the compositionality of large language models (LLMs), we introduce a new in-context learning approach to generate blob representations from text prompts. Our extensive experiments show that BlobGEN achieves superior zero-shot generation quality and better layout-guided controllability on MS-COCO. When augmented by LLMs, our method exhibits superior numerical and spatial correctness on compositional image generation benchmarks. Project page: https://blobgen-2d.github.io.