# UI Layout Generation with LLMs Guided by UI Grammar

arXiv: https://arxiv.org/abs/2310.15455 · HTML: https://mlsanity.com/paper/2310.15455/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators](https://mlsanity.com/paper/1901.06767/) (2019)
- [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020)
- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [Variational Transformer Networks for Layout Generation](https://mlsanity.com/paper/2104.02416/) (2021)
- [Neural Design Network: Graphic Layout Generation with Constraints](https://mlsanity.com/paper/1912.09421/) (2019)
- [Constrained Graphic Layout Generation via Latent Optimization](https://mlsanity.com/paper/2108.00871/) (2021)
- [BLT: Bidirectional Layout Transformer for Controllable Layout Generation](https://mlsanity.com/paper/2112.05112/) (2021)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [Unifying Layout Generation with a Decoupled Diffusion Model](https://mlsanity.com/paper/2303.05049/) (2023)
- [Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale](https://mlsanity.com/paper/2201.04100/) (2022)
- [PLay: Parametrically Conditioned Layout Generation using Latent Diffusion](https://mlsanity.com/paper/2301.11529/) (2023)
- [Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing](https://mlsanity.com/paper/2305.09434/) (2023)
- [Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoders](https://mlsanity.com/paper/2001.05308/) (2020)
- [Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models](https://mlsanity.com/paper/2110.07775/) (2021)
- [MenuCraft: Interactive Menu System Design with Large Language Models](https://mlsanity.com/paper/2303.04496/) (2023)

## Built on by (most cited first, 15 of 15)

- [Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers](https://mlsanity.com/paper/2505.21497/) (2025)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [Graphic Design with Large Multimodal Model](https://mlsanity.com/paper/2404.14368/) (2024)
- [ViMo: A Generative Visual GUI World Model for App Agents](https://mlsanity.com/paper/2504.13936/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Computer-Use Agents as Judges for Generative User Interface](https://mlsanity.com/paper/2511.15567/) (2025)
- [Generative Interfaces for Language Models](https://mlsanity.com/paper/2508.19227/) (2025)
- [Any2Poster: Any-Source Poster Generation Across Modalities and Domains](https://mlsanity.com/paper/2606.02915/) (2026)
- [CoGen: Creation of Reusable UI Components in Figma via Textual Commands](https://mlsanity.com/paper/2601.10536/) (2026)
- [GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication](https://mlsanity.com/paper/2502.10678/) (2025)
- [Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach](https://mlsanity.com/paper/2606.13192/) (2026)
- [Predicting the usability of mobile applications using AI tools: the rise of large user interface models, opportunities, and challenges](https://mlsanity.com/paper/2405.03716/) (2024)
- [FurniMAS: Language-Guided Furniture Decoration using Multi-Agent System](https://mlsanity.com/paper/2507.04770/) (2025)
- [APD-Agents: A Large Language Model-Driven Multi-Agents Collaborative Framework for Automated Page Design](https://mlsanity.com/paper/2511.14101/) (2025)
- [SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluation](https://mlsanity.com/paper/2608.22390/) (2026)

## Abstract (arXiv)

The recent advances in Large Language Models (LLMs) have stimulated interest among researchers and industry professionals, particularly in their application to tasks concerning mobile user interfaces (UIs). This position paper investigates the use of LLMs for UI layout generation. Central to our exploration is the introduction of UI grammar -- a novel approach we proposed to represent the hierarchical structure inherent in UI screens. The aim of this approach is to guide the generative capacities of LLMs more effectively and improve the explainability and controllability of the process. Initial experiments conducted with GPT-4 showed the promising capability of LLMs to produce high-quality user interfaces via in-context learning. Furthermore, our preliminary comparative study suggested the potential of the grammar-based approach in improving the quality of generative results in specific aspects.