# LayoutDM: Transformer-based Diffusion Model for Layout Generation

arXiv: https://arxiv.org/abs/2305.02567 · HTML: https://mlsanity.com/paper/2305.02567/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 36 of its 48 references that are in the map.
- First posted on arXiv 4 May 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [Generating Images with Sparse Representations](https://mlsanity.com/paper/2103.03841/) (2021)
- [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020)
- [LayoutVAE: Stochastic Scene Layout Generation From a Label Set](https://mlsanity.com/paper/1907.10719/) (2019)
- [Neural Photo Editing with Introspective Adversarial Networks](https://mlsanity.com/paper/1609.07093/) (2016)
- [CanvasVAE: Learning to Generate Vector Graphic Documents](https://mlsanity.com/paper/2108.01249/) (2021)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [Neural Design Network: Graphic Layout Generation with Constraints](https://mlsanity.com/paper/1912.09421/) (2019)
- [Constrained Graphic Layout Generation via Latent Optimization](https://mlsanity.com/paper/2108.00871/) (2021)

## Built on by (most cited first, 30 of 57)

- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation](https://mlsanity.com/paper/2502.18364/) (2025)
- [Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation](https://mlsanity.com/paper/2311.13602/) (2023)
- [PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation](https://mlsanity.com/paper/2404.00995/) (2024)
- [Mixed Diffusion for 3D Indoor Scene Synthesis](https://mlsanity.com/paper/2405.21066/) (2024)
- [UGG: Unified Generative Grasping](https://mlsanity.com/paper/2311.16917/) (2023)
- [Conditional Image Synthesis with Diffusion Models: A Survey](https://mlsanity.com/paper/2409.19365/) (2024)
- [Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints](https://mlsanity.com/paper/2402.04754/) (2024)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [LayoutFlow: Flow Matching for Layout Generation](https://mlsanity.com/paper/2403.18187/) (2024)
- [Alfie: Democratising RGBA Image Generation With No ](https://mlsanity.com/paper/2408.14826/) (2024)
- [LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models](https://mlsanity.com/paper/2309.09506/) (2023)
- [Planning and Rendering: Towards Product Poster Generation with Diffusion Models](https://mlsanity.com/paper/2312.08822/) (2023)
- [PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generation](https://mlsanity.com/paper/2505.07843/) (2025)
- [VASCAR: Content-Aware Layout Generation via Visual-Aware Self-Correction](https://mlsanity.com/paper/2412.04237/) (2024)
- [SciPostLayout: A Dataset for Layout Analysis and Layout Generation of Scientific Posters](https://mlsanity.com/paper/2407.19787/) (2024)
- [Training-free Composite Scene Generation for Layout-to-Image Synthesis](https://mlsanity.com/paper/2407.13609/) (2024)
- [TKG-DM: Training-free Chroma Key Content Generation Diffusion Model](https://mlsanity.com/paper/2411.15580/) (2024)
- [COHO: Context-Sensitive City-Scale Hierarchical Urban Layout Generation](https://mlsanity.com/paper/2407.11294/) (2024)
- [LTSim: Layout Transportation-based Similarity Measure for Evaluating Layout Generation](https://mlsanity.com/paper/2407.12356/) (2024)
- [Towards Better Text-to-Image Generation Alignment via Attention Modulation](https://mlsanity.com/paper/2404.13899/) (2024)
- [AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models](https://mlsanity.com/paper/2503.00591/) (2025)
- [Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model](https://mlsanity.com/paper/2409.16689/) (2024)
- [Lay-Your-Scene: Natural Scene Layout Generation with Diffusion Transformers](https://mlsanity.com/paper/2505.04718/) (2025)
- [CAL-RAG: Retrieval-Augmented Multi-Agent Generation for Content-Aware Layout Design](https://mlsanity.com/paper/2506.21934/) (2025)
- [InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Prior](https://mlsanity.com/paper/2407.07580/) (2024)
- [SemLayoutDiff: Semantic Layout Generation with Diffusion Model for Indoor Scene Synthesis](https://mlsanity.com/paper/2508.18597/) (2025)
- [Spatial Diffusion for Cell Layout Generation](https://mlsanity.com/paper/2409.03106/) (2024)
- [Smaller But Better: Unifying Layout Generation with Smaller Large Language Models](https://mlsanity.com/paper/2502.14005/) (2025)
- [Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Models](https://mlsanity.com/paper/2401.01130/) (2024)

## Abstract (arXiv)

Automatic layout generation that can synthesize high-quality layouts is an important tool for graphic design in many applications. Though existing methods based on generative models such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) have progressed, they still leave much room for improving the quality and diversity of the results. Inspired by the recent success of diffusion models in generating high-quality images, this paper explores their potential for conditional layout generation and proposes Transformer-based Layout Diffusion Model (LayoutDM) by instantiating the conditional denoising diffusion probabilistic model (DDPM) with a purely transformer-based architecture. Instead of using convolutional neural networks, a transformer-based conditional Layout Denoiser is proposed to learn the reverse diffusion process to generate samples from noised layout data. Benefitting from both transformer and DDPM, our LayoutDM is of desired properties such as high-quality generation, strong sample diversity, faithful distribution coverage, and stationary training in comparison to GANs and VAEs. Quantitative and qualitative experimental results show that our method outperforms state-of-the-art generative models in terms of quality and diversity.