# LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restriction

arXiv: https://arxiv.org/abs/2208.08037 · HTML: https://mlsanity.com/paper/2208.08037/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 17 Aug 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics](https://mlsanity.com/paper/2202.11705/) (2022)
- [NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics](https://mlsanity.com/paper/2112.08726/) (2021)
- [NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints](https://mlsanity.com/paper/2010.12884/) (2020)
- [LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators](https://mlsanity.com/paper/1901.06767/) (2019)
- [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020)
- [Guided Open Vocabulary Image Captioning with Constrained Beam Search](https://mlsanity.com/paper/1612.00576/) (2016)
- [CanvasVAE: Learning to Generate Vector Graphic Documents](https://mlsanity.com/paper/2108.01249/) (2021)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [Neural Design Network: Graphic Layout Generation with Constraints](https://mlsanity.com/paper/1912.09421/) (2019)
- [Constrained Graphic Layout Generation via Latent Optimization](https://mlsanity.com/paper/2108.00871/) (2021)
- [BLT: Bidirectional Layout Transformer for Controllable Layout Generation](https://mlsanity.com/paper/2112.05112/) (2021)
- [Attribute-conditioned Layout GAN for Automatic Graphic Design](https://mlsanity.com/paper/2009.05284/) (2020)
- [READ: Recursive Autoencoders for Document Layout Generation](https://mlsanity.com/paper/1909.00302/) (2019)
- [Few-Shot Semantic Parsing with Language Models Trained On Code](https://mlsanity.com/paper/2112.08696/) (2021)
- [Language Generation via Combinatorial Constraint Satisfaction: A Tree Search Enhanced Monte-Carlo Approach](https://mlsanity.com/paper/2011.12334/) (2020)

## Built on by (most cited first, 30 of 43)

- [DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception](https://mlsanity.com/paper/2410.12628/) (2024)
- [LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models](https://mlsanity.com/paper/2303.11589/) (2023)
- [PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM](https://mlsanity.com/paper/2406.02884/) (2024)
- [Graphic Design with Large Multimodal Model](https://mlsanity.com/paper/2404.14368/) (2024)
- [OpenCOLE: Towards Reproducible Automatic Graphic Design Generation](https://mlsanity.com/paper/2406.08232/) (2024)
- [ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation](https://mlsanity.com/paper/2502.18364/) (2025)
- [Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation](https://mlsanity.com/paper/2311.13602/) (2023)
- [PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation](https://mlsanity.com/paper/2404.00995/) (2024)
- [Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints](https://mlsanity.com/paper/2402.04754/) (2024)
- [Visual Text Generation in the Wild](https://mlsanity.com/paper/2407.14138/) (2024)
- [LayoutFlow: Flow Matching for Layout Generation](https://mlsanity.com/paper/2403.18187/) (2024)
- [CreatiPoster: Towards Editable and Controllable Multi-Layer Graphic Design Generation](https://mlsanity.com/paper/2506.10890/) (2025)
- [LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer](https://mlsanity.com/paper/2212.09877/) (2022)
- [VASCAR: Content-Aware Layout Generation via Visual-Aware Self-Correction](https://mlsanity.com/paper/2412.04237/) (2024)
- [SciPostLayout: A Dataset for Layout Analysis and Layout Generation of Scientific Posters](https://mlsanity.com/paper/2407.19787/) (2024)
- [From Elements to Design: A Layered Approach for Automatic Graphic Design Composition](https://mlsanity.com/paper/2412.19712/) (2024)
- [Towards Diverse and Consistent Typography Generation](https://mlsanity.com/paper/2309.02099/) (2023)
- [COHO: Context-Sensitive City-Scale Hierarchical Urban Layout Generation](https://mlsanity.com/paper/2407.11294/) (2024)
- [LTSim: Layout Transportation-based Similarity Measure for Evaluating Layout Generation](https://mlsanity.com/paper/2407.12356/) (2024)
- [TextLap: Customizing Language Models for Text-to-Layout Planning](https://mlsanity.com/paper/2410.12844/) (2024)
- [Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model](https://mlsanity.com/paper/2409.16689/) (2024)
- [Lay-Your-Scene: Natural Scene Layout Generation with Diffusion Transformers](https://mlsanity.com/paper/2505.04718/) (2025)
- [SEGA: A Stepwise Evolution Paradigm for Content-Aware Layout Generation with Design Prior](https://mlsanity.com/paper/2510.15749/) (2025)
- [DocSynthv2: A Practical Autoregressive Modeling for Document Generation](https://mlsanity.com/paper/2406.08354/) (2024)
- [Smaller But Better: Unifying Layout Generation with Smaller Large Language Models](https://mlsanity.com/paper/2502.14005/) (2025)
- [Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Models](https://mlsanity.com/paper/2401.01130/) (2024)
- [GraphicBench: A Planning Benchmark for Graphic Design with Language Agents](https://mlsanity.com/paper/2504.11571/) (2025)
- [SciPostGen: Bridging the Gap between Scientific Papers and Poster Layouts](https://mlsanity.com/paper/2511.22490/) (2025)
- [Beyond Human Annotation: Recent Advances in Data Generation Methods for Document Intelligence](https://mlsanity.com/paper/2601.12318/) (2026)
- [GLDesigner: Leveraging Multi-Modal LLMs as Designer for Enhanced Aesthetic Text Glyph Layouts](https://mlsanity.com/paper/2411.11435/) (2024)

## Abstract (arXiv)

Conditional graphic layout generation, which generates realistic layouts according to user constraints, is a challenging task that has not been well-studied yet. First, there is limited discussion about how to handle diverse user constraints flexibly and uniformly. Second, to make the layouts conform to user constraints, existing work often sacrifices generation quality significantly. In this work, we propose LayoutFormer++ to tackle the above problems. First, to flexibly handle diverse constraints, we propose a constraint serialization scheme, which represents different user constraints as sequences of tokens with a predefined format. Then, we formulate conditional layout generation as a sequence-to-sequence transformation, and leverage encoder-decoder framework with Transformer as the basic architecture. Furthermore, to make the layout better meet user requirements without harming quality, we propose a decoding space restriction strategy. Specifically, we prune the predicted distribution by ignoring the options that definitely violate user constraints and likely result in low-quality layouts, and make the model samples from the restricted distribution. Experiments demonstrate that LayoutFormer++ outperforms existing approaches on all the tasks in terms of both better generation quality and less constraint violation.