# Boosting GUI Prototyping with Diffusion Models

arXiv: https://arxiv.org/abs/2306.06233 · HTML: https://mlsanity.com/paper/2306.06233/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [LayoutDM: Discrete Diffusion Model for Controllable Layout Generation](https://mlsanity.com/paper/2303.08137/) (2023)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [Unifying Layout Generation with a Decoupled Diffusion Model](https://mlsanity.com/paper/2303.05049/) (2023)
- [VINS: Visual Search for Mobile User Interface Design](https://mlsanity.com/paper/2102.05216/) (2021)
- [PLay: Parametrically Conditioned Layout Generation using Latent Diffusion](https://mlsanity.com/paper/2301.11529/) (2023)
- [DLT: Conditioned layout generation with Joint Discrete-Continuous Diffusion Layout Transformer](https://mlsanity.com/paper/2303.03755/) (2023)
- [GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks](https://mlsanity.com/paper/2101.09978/) (2021)
- [Wireframe-Based UI Design Search Through Image Autoencoder](https://mlsanity.com/paper/2103.07085/) (2021)
- [Automated Reporting of GUI Design Violations for Mobile Apps](https://mlsanity.com/paper/1802.04732/) (2018)
- [Gallery D.C.: Auto-created GUI Component Gallery for Design Search and Knowledge Discovery](https://mlsanity.com/paper/2204.06700/) (2022)
- [Guigle: A GUI Search Engine for Android Apps](https://mlsanity.com/paper/1901.00891/) (2019)

## Built on by (most cited first, 10 of 10)

- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Design](https://mlsanity.com/paper/2311.16974/) (2023)
- [Graphic Design with Large Multimodal Model](https://mlsanity.com/paper/2404.14368/) (2024)
- [Code2World: A GUI World Model via Renderable Code Generation](https://mlsanity.com/paper/2602.09856/) (2026)
- [GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environments](https://mlsanity.com/paper/2602.09007/) (2026)
- [GUing: A Mobile GUI Search Engine using a Vision-Language Model](https://mlsanity.com/paper/2405.00145/) (2024)
- [On AI-Inspired UI-Design](https://mlsanity.com/paper/2406.13631/) (2024)
- [Prompt-to-OS (P2OS): Revolutionizing Operating Systems and Human-Computer Interaction with Integrated AI Generative Models](https://mlsanity.com/paper/2310.04875/) (2023)
- [CoGen: Creation of Reusable UI Components in Figma via Textual Commands](https://mlsanity.com/paper/2601.10536/) (2026)
- [Bridging Gulfs in UI Generation through Semantic Guidance](https://mlsanity.com/paper/2601.19171/) (2026)

## Abstract (arXiv)

GUI (graphical user interface) prototyping is a widely-used technique in requirements engineering for gathering and refining requirements, reducing development risks and increasing stakeholder engagement. However, GUI prototyping can be a time-consuming and costly process. In recent years, deep learning models such as Stable Diffusion have emerged as a powerful text-to-image tool capable of generating detailed images based on text prompts. In this paper, we propose UI-Diffuser, an approach that leverages Stable Diffusion to generate mobile UIs through simple textual descriptions and UI components. Preliminary results show that UI-Diffuser provides an efficient and cost-effective way to generate mobile GUI designs while reducing the need for extensive prototyping efforts. This approach has the potential to significantly improve the speed and efficiency of GUI prototyping in requirements engineering.