# A Parse-Then-Place Approach for Generating Graphic Layouts from Textual Descriptions

arXiv: https://arxiv.org/abs/2308.12700 · HTML: https://mlsanity.com/paper/2308.12700/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators](https://mlsanity.com/paper/1901.06767/) (2019)
- [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020)
- [LayoutVAE: Stochastic Scene Layout Generation From a Label Set](https://mlsanity.com/paper/1907.10719/) (2019)
- [Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis](https://mlsanity.com/paper/1801.05091/) (2018)
- [Semantic Parsing for Task Oriented Dialog using Hierarchical Representations](https://mlsanity.com/paper/1810.07942/) (2018)
- [UIBert: Learning Generic Multimodal Representations for UI Understanding](https://mlsanity.com/paper/2107.13731/) (2021)
- [CanvasVAE: Learning to Generate Vector Graphic Documents](https://mlsanity.com/paper/2108.01249/) (2021)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [Neural Design Network: Graphic Layout Generation with Constraints](https://mlsanity.com/paper/1912.09421/) (2019)
- [Constrained Graphic Layout Generation via Latent Optimization](https://mlsanity.com/paper/2108.00871/) (2021)
- [BLT: Bidirectional Layout Transformer for Controllable Layout Generation](https://mlsanity.com/paper/2112.05112/) (2021)
- [Attribute-conditioned Layout GAN for Automatic Graphic Design](https://mlsanity.com/paper/2009.05284/) (2020)
- [Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Tracking](https://mlsanity.com/paper/2105.04222/) (2021)
- [Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsing](https://mlsanity.com/paper/2001.11458/) (2020)
- [Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations](https://mlsanity.com/paper/2104.07478/) (2021)
- [READ: Recursive Autoencoders for Document Layout Generation](https://mlsanity.com/paper/1909.00302/) (2019)
- [The Power of Prompt Tuning for Low-Resource Semantic Parsing](https://mlsanity.com/paper/2110.08525/) (2021)
- [Intelligent Home 3D: Automatic 3D-House Design from Linguistic Descriptions Only](https://mlsanity.com/paper/2003.00397/) (2020)
- [Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models](https://mlsanity.com/paper/2110.07775/) (2021)

## Built on by (most cited first, 15 of 15)

- [If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents](https://mlsanity.com/paper/2401.00812/) (2024)
- [LayoutPrompter: Awaken the Design Ability of Large Language Models](https://mlsanity.com/paper/2311.06495/) (2023)
- [PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM](https://mlsanity.com/paper/2406.02884/) (2024)
- [Graphic Design with Large Multimodal Model](https://mlsanity.com/paper/2404.14368/) (2024)
- [Multimodal Markup Document Models for Graphic Design Completion](https://mlsanity.com/paper/2409.19051/) (2024)
- [A Large Scale Analysis of Gender Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2503.23398/) (2025)
- [From Elements to Design: A Layered Approach for Automatic Graphic Design Composition](https://mlsanity.com/paper/2412.19712/) (2024)
- [LayoutCoT: Unleashing the Deep Reasoning Potential of Large Language Models for Layout Generation](https://mlsanity.com/paper/2504.10829/) (2025)
- [Smaller But Better: Unifying Layout Generation with Smaller Large Language Models](https://mlsanity.com/paper/2502.14005/) (2025)
- [Revision Matters: Generative Design Guided by Revision Edits](https://mlsanity.com/paper/2406.18559/) (2024)
- [GLDesigner: Leveraging Multi-Modal LLMs as Designer for Enhanced Aesthetic Text Glyph Layouts](https://mlsanity.com/paper/2411.11435/) (2024)
- [Generating Animated Layouts as Structured Text Representations](https://mlsanity.com/paper/2505.00975/) (2025)
- [StructLayoutFormer:Conditional Structured Layout Generation via Structure Serialization and Disentanglement](https://mlsanity.com/paper/2510.26141/) (2025)
- [Sketch-to-Layout: Sketch-Guided Multimodal Layout Generation](https://mlsanity.com/paper/2510.27632/) (2025)
- [ReLayout: Versatile and Structure-Preserving Design Layout Editing via Relation-Aware Design Reconstruction](https://mlsanity.com/paper/2602.01046/) (2026)

## Abstract (arXiv)

Creating layouts is a fundamental step in graphic design. In this work, we propose to use text as the guidance to create graphic layouts, i.e., Text-to-Layout, aiming to lower the design barriers. Text-to-Layout is a challenging task, because it needs to consider the implicit, combined, and incomplete layout constraints from text, each of which has not been studied in previous work. To address this, we present a two-stage approach, named parse-then-place. The approach introduces an intermediate representation (IR) between text and layout to represent diverse layout constraints. With IR, Text-to-Layout is decomposed into a parse stage and a place stage. The parse stage takes a textual description as input and generates an IR, in which the implicit constraints from the text are transformed into explicit ones. The place stage generates layouts based on the IR. To model combined and incomplete constraints, we use a Transformer-based layout generation model and carefully design a way to represent constraints and layouts as sequences. Besides, we adopt the pretrain-then-finetune strategy to boost the performance of the layout generation model with large-scale unlabeled layouts. To evaluate our approach, we construct two Text-to-Layout datasets and conduct experiments on them. Quantitative results, qualitative analysis, and user studies demonstrate the effectiveness of our approach.