# PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generation

arXiv: https://arxiv.org/abs/2404.00995 · HTML: https://mlsanity.com/paper/2404.00995/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 52 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 30 of 41)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)
- [AnyText: Multilingual Visual Text Generation And Editing](https://mlsanity.com/paper/2311.03054/) (2023)
- [LayoutDM: Discrete Diffusion Model for Controllable Layout Generation](https://mlsanity.com/paper/2303.08137/) (2023)
- [LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators](https://mlsanity.com/paper/1901.06767/) (2019)
- [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020)
- [LayoutVAE: Stochastic Scene Layout Generation From a Label Set](https://mlsanity.com/paper/1907.10719/) (2019)
- [CanvasVAE: Learning to Generate Vector Graphic Documents](https://mlsanity.com/paper/2108.01249/) (2021)
- [Character-Aware Models Improve Visual Text Rendering](https://mlsanity.com/paper/2212.10562/) (2022)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [Constrained Graphic Layout Generation via Latent Optimization](https://mlsanity.com/paper/2108.00871/) (2021)
- [LLM-grounded Video Diffusion Models](https://mlsanity.com/paper/2309.17444/) (2023)
- [BLT: Bidirectional Layout Transformer for Controllable Layout Generation](https://mlsanity.com/paper/2112.05112/) (2021)
- [Attribute-conditioned Layout GAN for Automatic Graphic Design](https://mlsanity.com/paper/2009.05284/) (2020)
- [PosterLayout: A New Benchmark and Approach for Content-aware Visual-Textual Presentation Layout](https://mlsanity.com/paper/2303.15937/) (2023)
- [Composition-aware Graphic Layout GAN for Visual-textual Presentation Designs](https://mlsanity.com/paper/2205.00303/) (2022)
- [LayoutDM: Transformer-based Diffusion Model for Layout Generation](https://mlsanity.com/paper/2305.02567/) (2023)

## Built on by (most cited first, 30 of 35)

- [PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Framework](https://mlsanity.com/paper/2506.10741/) (2025)
- [CreatiPoster: Towards Editable and Controllable Multi-Layer Graphic Design Generation](https://mlsanity.com/paper/2506.10890/) (2025)
- [Multimodal Markup Document Models for Graphic Design Completion](https://mlsanity.com/paper/2409.19051/) (2024)
- [PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generation](https://mlsanity.com/paper/2505.07843/) (2025)
- [LLM/Agent-as-Data-Analyst: A Survey](https://mlsanity.com/paper/2509.23988/) (2025)
- [VASCAR: Content-Aware Layout Generation via Visual-Aware Self-Correction](https://mlsanity.com/paper/2412.04237/) (2024)
- [Uni-Layout: Integrating Human Feedback in Unified Layout Generation and Evaluation](https://mlsanity.com/paper/2508.02374/) (2025)
- [From Elements to Design: A Layered Approach for Automatic Graphic Design Composition](https://mlsanity.com/paper/2412.19712/) (2024)
- [PosterGen: Aesthetic-Aware Multi-Modal Paper-to-Poster Generation via Multi-Agent LLMs](https://mlsanity.com/paper/2508.17188/) (2025)
- [Rethinking Layered Graphic Design Generation with a Top-Down Approach](https://mlsanity.com/paper/2507.05601/) (2025)
- [AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models](https://mlsanity.com/paper/2503.00591/) (2025)
- [LayoutCoT: Unleashing the Deep Reasoning Potential of Large Language Models for Layout Generation](https://mlsanity.com/paper/2504.10829/) (2025)
- [SEGA: A Stepwise Evolution Paradigm for Content-Aware Layout Generation with Design Prior](https://mlsanity.com/paper/2510.15749/) (2025)
- [T-Stars-Poster: A Framework for Product-Centric Advertising Image Design](https://mlsanity.com/paper/2501.14316/) (2025)
- [DesignAsCode: Bridging Structural Editability and Visual Fidelity in Graphic Design Generation](https://mlsanity.com/paper/2602.17690/) (2026)
- [SciPostGen: Bridging the Gap between Scientific Papers and Poster Layouts](https://mlsanity.com/paper/2511.22490/) (2025)
- [Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement](https://mlsanity.com/paper/2603.22187/) (2026)
- [PRISM: Privacy-Preserving Improved Stochastic Masking for Federated Generative Models](https://mlsanity.com/paper/2503.08085/) (2025)
- [Improving User Interface Generation Models from Designer Feedback](https://mlsanity.com/paper/2509.16779/) (2025)
- [Scan-and-Print: Patch-level Data Summarization and Augmentation for Content-aware Layout Generation in Poster Design](https://mlsanity.com/paper/2505.20649/) (2025)
- [PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow](https://mlsanity.com/paper/2603.25738/) (2026)
- [SceneVTG++: Controllable Multilingual Visual Text Generation in the Wild](https://mlsanity.com/paper/2501.02962/) (2025)
- [AnyDoc: Enhancing Document Generation via Large-Scale HTML/CSS Data Synthesis and Height-Aware Reinforcement Optimization](https://mlsanity.com/paper/2603.25118/) (2026)
- [Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design](https://mlsanity.com/paper/2609.22086/) (2026)
- [Aggregated Structural Representation with Large Language Models for Human-Centric Layout Generation](https://mlsanity.com/paper/2505.19554/) (2025)
- [Automatic Text Box Placement for Supporting Typographic Design](https://mlsanity.com/paper/2510.07665/) (2025)
- [Sketch-to-Layout: Sketch-Guided Multimodal Layout Generation](https://mlsanity.com/paper/2510.27632/) (2025)
- [UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generation](https://mlsanity.com/paper/2512.08897/) (2025)
- [Content-Aware Ad Banner Layout Generation with Two-Stage Chain-of-Thought in Vision Language Models](https://mlsanity.com/paper/2512.12596/) (2025)
- [PRISM: Learning Design Knowledge from Data for Stylistic Design Improvement](https://mlsanity.com/paper/2601.11747/) (2026)

## Abstract (arXiv)

Visual layout plays a critical role in graphic design fields such as advertising, posters, and web UI design. The recent trend towards content-aware layout generation through generative models has shown promise, yet it often overlooks the semantic intricacies of layout design by treating it as a simple numerical optimization. To bridge this gap, we introduce PosterLlama, a network designed for generating visually and textually coherent layouts by reformatting layout elements into HTML code and leveraging the rich design knowledge embedded within language models. Furthermore, we enhance the robustness of our model with a unique depth-based poster augmentation strategy. This ensures our generated layouts remain semantically rich but also visually appealing, even with limited data. Our extensive evaluations across several benchmarks demonstrate that PosterLlama outperforms existing methods in producing authentic and content-aware layouts. It supports an unparalleled range of conditions, including but not limited to unconditional layout generation, element conditional layout generation, layout completion, among others, serving as a highly versatile user manipulation tool.