# LayoutTransformer: Layout Generation and Completion with Self-attention

arXiv: https://arxiv.org/abs/2006.14615 · HTML: https://mlsanity.com/paper/2006.14615/

## Key facts

- Cited by 123 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2020.

## Built on by (most cited first, 30 of 123)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models](https://mlsanity.com/paper/2305.13655/) (2023)
- [LayoutDM: Discrete Diffusion Model for Controllable Layout Generation](https://mlsanity.com/paper/2303.08137/) (2023)
- [TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering](https://mlsanity.com/paper/2311.16465/) (2023)
- [LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation](https://mlsanity.com/paper/2308.05095/) (2023)
- [DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception](https://mlsanity.com/paper/2410.12628/) (2024)
- [TaleCrafter: Interactive Story Visualization with Multiple Characters](https://mlsanity.com/paper/2305.18247/) (2023)
- [BLT: Bidirectional Layout Transformer for Controllable Layout Generation](https://mlsanity.com/paper/2112.05112/) (2021)
- [DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis](https://mlsanity.com/paper/2212.11984/) (2022)
- [Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraints](https://mlsanity.com/paper/2310.03602/) (2023)
- [LayoutDM: Transformer-based Diffusion Model for Layout Generation](https://mlsanity.com/paper/2305.02567/) (2023)
- [LayoutPrompter: Awaken the Design Ability of Large Language Models](https://mlsanity.com/paper/2311.06495/) (2023)
- [LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restriction](https://mlsanity.com/paper/2208.08037/) (2022)
- [Unifying Layout Generation with a Decoupled Diffusion Model](https://mlsanity.com/paper/2303.05049/) (2023)
- [LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models](https://mlsanity.com/paper/2303.11589/) (2023)
- [PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM](https://mlsanity.com/paper/2406.02884/) (2024)
- [OpenCOLE: Towards Reproducible Automatic Graphic Design Generation](https://mlsanity.com/paper/2406.08232/) (2024)
- [Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation](https://mlsanity.com/paper/2311.13602/) (2023)
- [PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation](https://mlsanity.com/paper/2404.00995/) (2024)
- [Geometry Aligned Variational Transformer for Image-conditioned Layout Generation](https://mlsanity.com/paper/2209.00852/) (2022)
- [Towards Flexible Multi-modal Document Models](https://mlsanity.com/paper/2303.18248/) (2023)
- [Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints](https://mlsanity.com/paper/2402.04754/) (2024)
- [NAP: Neural 3D Articulation Prior](https://mlsanity.com/paper/2305.16315/) (2023)
- [3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs](https://mlsanity.com/paper/2209.07896/) (2022)
- [PLay: Parametrically Conditioned Layout Generation using Latent Diffusion](https://mlsanity.com/paper/2301.11529/) (2023)
- [LANS: A Layout-Aware Neural Solver for Plane Geometry Problem](https://mlsanity.com/paper/2311.16476/) (2023)
- [Dolfin: Diffusion Layout Transformers without Autoencoder](https://mlsanity.com/paper/2310.16305/) (2023)
- [Diffusion-based Document Layout Generation](https://mlsanity.com/paper/2303.10787/) (2023)
- [LayoutFlow: Flow Matching for Layout Generation](https://mlsanity.com/paper/2403.18187/) (2024)

## Abstract (arXiv)

We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents, and 3D objects. Most complex scenes, natural or human-designed, can be expressed as a meaningful arrangement of simpler compositional graphical primitives. Generating a new layout or extending an existing layout requires understanding the relationships between these primitives. To do this, we propose LayoutTransformer, a novel framework that leverages self-attention to learn contextual relationships between layout elements and generate novel layouts in a given domain. Our framework allows us to generate a new layout either from an empty set or from an initial seed set of primitives, and can easily scale to support an arbitrary of primitives per layout. Furthermore, our analyses show that the model is able to automatically capture the semantic properties of the primitives. We propose simple improvements in both representation of layout primitives, as well as training methods to demonstrate competitive performance in very diverse data domains such as object bounding boxes in natural images(COCO bounding box), documents (PubLayNet), mobile applications (RICO dataset) as well as 3D shapes (Part-Net). Code and other materials will be made available at https://kampta.github.io/layout.