# Geometry Aligned Variational Transformer for Image-conditioned Layout Generation

arXiv: https://arxiv.org/abs/2209.00852 · HTML: https://mlsanity.com/paper/2209.00852/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 2 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing](https://mlsanity.com/paper/1903.10145/) (2019)
- [LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators](https://mlsanity.com/paper/1901.06767/) (2019)
- [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020)
- [LayoutVAE: Stochastic Scene Layout Generation From a Label Set](https://mlsanity.com/paper/1907.10719/) (2019)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [An Uncertain Future: Forecasting from Static Images using Variational Autoencoders](https://mlsanity.com/paper/1606.07873/) (2016)
- [Neural Design Network: Graphic Layout Generation with Constraints](https://mlsanity.com/paper/1912.09421/) (2019)
- [Attribute-conditioned Layout GAN for Automatic Graphic Design](https://mlsanity.com/paper/2009.05284/) (2020)
- [Normalized and Geometry-Aware Self-Attention Network for Image Captioning](https://mlsanity.com/paper/2003.08897/) (2020)
- [Information Constraints on Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1805.08672/) (2018)

## Built on by (most cited first, 30 of 33)

- [PosterLayout: A New Benchmark and Approach for Content-aware Visual-Textual Presentation Layout](https://mlsanity.com/paper/2303.15937/) (2023)
- [LayoutDM: Transformer-based Diffusion Model for Layout Generation](https://mlsanity.com/paper/2305.02567/) (2023)
- [AutoPoster: A Highly Automatic and Content-aware Design System for Advertising Poster Generation](https://mlsanity.com/paper/2308.01095/) (2023)
- [PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM](https://mlsanity.com/paper/2406.02884/) (2024)
- [Relation-Aware Diffusion Model for Controllable Poster Layout Generation](https://mlsanity.com/paper/2306.09086/) (2023)
- [Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation](https://mlsanity.com/paper/2311.13602/) (2023)
- [PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation](https://mlsanity.com/paper/2404.00995/) (2024)
- [PLay: Parametrically Conditioned Layout Generation using Latent Diffusion](https://mlsanity.com/paper/2301.11529/) (2023)
- [Tell2Design: A Dataset for Language-Guided Floor Plan Generation](https://mlsanity.com/paper/2311.15941/) (2023)
- [Desigen: A Pipeline for Controllable Design Template Generation](https://mlsanity.com/paper/2403.09093/) (2024)
- [TextPainter: Multimodal Text Image Generation with Visual-harmony and Text-comprehension for Poster Design](https://mlsanity.com/paper/2308.04733/) (2023)
- [Planning and Rendering: Towards Product Poster Generation with Diffusion Models](https://mlsanity.com/paper/2312.08822/) (2023)
- [PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generation](https://mlsanity.com/paper/2505.07843/) (2025)
- [LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer](https://mlsanity.com/paper/2212.09877/) (2022)
- [VASCAR: Content-Aware Layout Generation via Visual-Aware Self-Correction](https://mlsanity.com/paper/2412.04237/) (2024)
- [Unsupervised Domain Adaption with Pixel-level Discriminator for Image-aware Layout Generation](https://mlsanity.com/paper/2303.14377/) (2023)
- [Towards Diverse and Consistent Typography Generation](https://mlsanity.com/paper/2309.02099/) (2023)
- [AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models](https://mlsanity.com/paper/2503.00591/) (2025)
- [Constrained Layout Generation with Factor Graphs](https://mlsanity.com/paper/2404.00385/) (2024)
- [InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Prior](https://mlsanity.com/paper/2407.07580/) (2024)
- [SEGA: A Stepwise Evolution Paradigm for Content-Aware Layout Generation with Design Prior](https://mlsanity.com/paper/2510.15749/) (2025)
- [Spot the Error: Non-autoregressive Graphic Layout Generation with Wireframe Locator](https://mlsanity.com/paper/2401.16375/) (2024)
- [CoLay: Controllable Layout Generation through Multi-conditional Latent Diffusion](https://mlsanity.com/paper/2405.13045/) (2024)
- [GLDesigner: Leveraging Multi-Modal LLMs as Designer for Enhanced Aesthetic Text Glyph Layouts](https://mlsanity.com/paper/2411.11435/) (2024)
- [Scan-and-Print: Patch-level Data Summarization and Augmentation for Content-aware Layout Generation in Poster Design](https://mlsanity.com/paper/2505.20649/) (2025)
- [PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow](https://mlsanity.com/paper/2603.25738/) (2026)
- [LayoutDiT: Exploring Content-Graphic Balance in Layout Generation with Diffusion Transformer](https://mlsanity.com/paper/2407.15233/) (2024)
- [LLMs as Layout Designers: Enhanced Spatial Reasoning for Content-Aware Layout Generation](https://mlsanity.com/paper/2509.16891/) (2025)
- [UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generation](https://mlsanity.com/paper/2512.08897/) (2025)
- [DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation](https://mlsanity.com/paper/2602.23438/) (2026)

## Abstract (arXiv)

Layout generation is a novel task in computer vision, which combines the challenges in both object localization and aesthetic appraisal, widely used in advertisements, posters, and slides design. An accurate and pleasant layout should consider both the intra-domain relationship within layout elements and the inter-domain relationship between layout elements and the image. However, most previous methods simply focus on image-content-agnostic layout generation, without leveraging the complex visual information from the image. To this end, we explore a novel paradigm entitled image-conditioned layout generation, which aims to add text overlays to an image in a semantically coherent manner. Specifically, we propose an Image-Conditioned Variational Transformer (ICVT) that autoregressively generates various layouts in an image. First, self-attention mechanism is adopted to model the contextual relationship within layout elements, while cross-attention mechanism is used to fuse the visual information of conditional images. Subsequently, we take them as building blocks of conditional variational autoencoder (CVAE), which demonstrates appealing diversity. Second, in order to alleviate the gap between layout elements domain and visual domain, we design a Geometry Alignment module, in which the geometric information of the image is aligned with the layout representation. In addition, we construct a large-scale advertisement poster layout designing dataset with delicate layout and saliency map annotations. Experimental results show that our model can adaptively generate layouts in the non-intrusive area of the image, resulting in a harmonious layout design.