# Unifying Layout Generation with a Decoupled Diffusion Model

arXiv: https://arxiv.org/abs/2303.05049 · HTML: https://mlsanity.com/paper/2303.05049/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 24 of 24)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators](https://mlsanity.com/paper/1901.06767/) (2019)
- [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020)
- [LayoutVAE: Stochastic Scene Layout Generation From a Label Set](https://mlsanity.com/paper/1907.10719/) (2019)
- [CanvasVAE: Learning to Generate Vector Graphic Documents](https://mlsanity.com/paper/2108.01249/) (2021)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [Neural Design Network: Graphic Layout Generation with Constraints](https://mlsanity.com/paper/1912.09421/) (2019)
- [Constrained Graphic Layout Generation via Latent Optimization](https://mlsanity.com/paper/2108.00871/) (2021)
- [BLT: Bidirectional Layout Transformer for Controllable Layout Generation](https://mlsanity.com/paper/2112.05112/) (2021)
- [Attribute-conditioned Layout GAN for Automatic Graphic Design](https://mlsanity.com/paper/2009.05284/) (2020)
- [READ: Recursive Autoencoders for Document Layout Generation](https://mlsanity.com/paper/1909.00302/) (2019)

## Built on by (most cited first, 30 of 41)

- [DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception](https://mlsanity.com/paper/2410.12628/) (2024)
- [LayoutPrompter: Awaken the Design Ability of Large Language Models](https://mlsanity.com/paper/2311.06495/) (2023)
- [PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM](https://mlsanity.com/paper/2406.02884/) (2024)
- [Relation-Aware Diffusion Model for Controllable Poster Layout Generation](https://mlsanity.com/paper/2306.09086/) (2023)
- [ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation](https://mlsanity.com/paper/2502.18364/) (2025)
- [PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation](https://mlsanity.com/paper/2404.00995/) (2024)
- [Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints](https://mlsanity.com/paper/2402.04754/) (2024)
- [LayoutFlow: Flow Matching for Layout Generation](https://mlsanity.com/paper/2403.18187/) (2024)
- [Alfie: Democratising RGBA Image Generation With No ](https://mlsanity.com/paper/2408.14826/) (2024)
- [LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models](https://mlsanity.com/paper/2309.09506/) (2023)
- [Planning and Rendering: Towards Product Poster Generation with Diffusion Models](https://mlsanity.com/paper/2312.08822/) (2023)
- [Multimodal Markup Document Models for Graphic Design Completion](https://mlsanity.com/paper/2409.19051/) (2024)
- [LLM/Agent-as-Data-Analyst: A Survey](https://mlsanity.com/paper/2509.23988/) (2025)
- [UI Layout Generation with LLMs Guided by UI Grammar](https://mlsanity.com/paper/2310.15455/) (2023)
- [SciPostLayout: A Dataset for Layout Analysis and Layout Generation of Scientific Posters](https://mlsanity.com/paper/2407.19787/) (2024)
- [Boosting GUI Prototyping with Diffusion Models](https://mlsanity.com/paper/2306.06233/) (2023)
- [COHO: Context-Sensitive City-Scale Hierarchical Urban Layout Generation](https://mlsanity.com/paper/2407.11294/) (2024)
- [LTSim: Layout Transportation-based Similarity Measure for Evaluating Layout Generation](https://mlsanity.com/paper/2407.12356/) (2024)
- [Towards Better Text-to-Image Generation Alignment via Attention Modulation](https://mlsanity.com/paper/2404.13899/) (2024)
- [AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models](https://mlsanity.com/paper/2503.00591/) (2025)
- [Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model](https://mlsanity.com/paper/2409.16689/) (2024)
- [LayoutCoT: Unleashing the Deep Reasoning Potential of Large Language Models for Layout Generation](https://mlsanity.com/paper/2504.10829/) (2025)
- [DocSynthv2: A Practical Autoregressive Modeling for Document Generation](https://mlsanity.com/paper/2406.08354/) (2024)
- [Smaller But Better: Unifying Layout Generation with Smaller Large Language Models](https://mlsanity.com/paper/2502.14005/) (2025)
- [Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Models](https://mlsanity.com/paper/2401.01130/) (2024)
- [WebRPG: Automatic Web Rendering Parameters Generation for Visual Presentation](https://mlsanity.com/paper/2407.15502/) (2024)
- [Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligence](https://mlsanity.com/paper/2601.12318/) (2026)
- [TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generation](https://mlsanity.com/paper/2404.11824/) (2024)
- [GLDesigner: Leveraging Multi-Modal LLMs as Designer for Enhanced Aesthetic Text Glyph Layouts](https://mlsanity.com/paper/2411.11435/) (2024)
- [LayoutRAG: Retrieval-Augmented Model for Content-agnostic Conditional Layout Generation](https://mlsanity.com/paper/2506.02697/) (2025)

## Abstract (arXiv)

Layout generation aims to synthesize realistic graphic scenes consisting of elements with different attributes including category, size, position, and between-element relation. It is a crucial task for reducing the burden on heavy-duty graphic design works for formatted scenes, e.g., publications, documents, and user interfaces (UIs). Diverse application scenarios impose a big challenge in unifying various layout generation subtasks, including conditional and unconditional generation. In this paper, we propose a Layout Diffusion Generative Model (LDGM) to achieve such unification with a single decoupled diffusion model. LDGM views a layout of arbitrary missing or coarse element attributes as an intermediate diffusion status from a completed layout. Since different attributes have their individual semantics and characteristics, we propose to decouple the diffusion processes for them to improve the diversity of training samples and learn the reverse process jointly to exploit global-scope contexts for facilitating generation. As a result, our LDGM can generate layouts either from scratch or conditional on arbitrary available attributes. Extensive qualitative and quantitative experiments demonstrate our proposed LDGM outperforms existing layout generation models in both functionality and performance.