Boosting GUI Prototyping with Diffusion Models
On the map: Diffusion models → Image generation, text-to-image and editing
Abstract (arXiv)
GUI (graphical user interface) prototyping is a widely-used technique in requirements engineering for gathering and refining requirements, reducing development risks and increasing stakeholder engagement. However, GUI prototyping can be a time-consuming and costly process. In recent years, deep learning models such as Stable Diffusion have emerged as a powerful text-to-image tool capable of generating detailed images based on text prompts. In this paper, we propose UI-Diffuser, an approach that leverages Stable Diffusion to generate mobile UIs through simple textual descriptions and UI components. Preliminary results show that UI-Diffuser provides an efficient and cost-effective way to generate mobile GUI designs while reducing the need for extensive prototyping efforts. This approach has the potential to significantly improve the speed and efficiency of GUI prototyping in requirements engineering.
Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 25 of its 44 references that are in the map. First posted on arXiv 9 Jun 2023.
Builds on
25 of 44 references.
Closest ancestors · 8
Same direction, newest first
- 2023LayoutDM: Discrete Diffusion Model for Controllable Layout Generation14 Mar
- 2023Unifying Layout Generation with a Decoupled Diffusion Model9 Mar
- 2023DLT: Conditioned layout generation with Joint Discrete-Continuous Diffusion Layout Transformer7 Mar
- 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
- 2023PLay: Parametrically Conditioned Layout Generation using Latent Diffusion27 Jan
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
Other references · 17
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
- 2022Generative Adversarial Networkscited by 5,603
- 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
- 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
- 2021Screen2Vec: Semantic Embedding of GUI Screens and GUI Componentscited by 41
- 2021VINS: Visual Search for Mobile User Interface Designcited by 37
5 more
- 2021GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networkscited by 17
- 2021Wireframe-Based UI Design Search Through Image Autoencodercited by 17
- 2018Automated Reporting of GUI Design Violations for Mobile Appscited by 15
- 2022Gallery D.C.: Auto-created GUI Component Gallery for Design Search and Knowledge Discoverycited by 5
- 2019Guigle: A GUI Search Engine for Android Appscited by 4
Built on by
Citing papers per year: 2023: 3; 2024: 3; 2025: 0; 2026: 4.
10 papers.
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2023COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Designcited by 39
- 2024Graphic Design with Large Multimodal Modelcited by 38
- 2026Code2World: A GUI World Model via Renderable Code Generationcited by 26
- 2026GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environmentscited by 5
- 2024GUing: A Mobile GUI Search Engine using a Vision-Language Modelcited by 4
- 2024On AI-Inspired UI-Designcited by 4
- 2023Prompt-to-OS (P2OS): Revolutionizing Operating Systems and Human-Computer Interaction with Integrated AI Generative Modelscited by 3
- 2026CoGen: Creation of Reusable UI Components in Figma via Textual Commandscited by 2
- 2026Bridging Gulfs in UI Generation through Semantic Guidancecited by 1
- 2026Code2World: A GUI World Model via Renderable Code Generation10 Feb 2026
- 2026GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environments9 Feb 2026
- 2026Bridging Gulfs in UI Generation through Semantic Guidance27 Jan 2026
- 2026CoGen: Creation of Reusable UI Components in Figma via Textual Commands15 Jan 2026
- 2024On AI-Inspired UI-Design19 Jun 2024
- 2024GUing: A Mobile GUI Search Engine using a Vision-Language Model30 Apr 2024
- 2024Graphic Design with Large Multimodal Model22 Apr 2024
- 2023COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Design28 Nov 2023
- 2023Prompt-to-OS (P2OS): Revolutionizing Operating Systems and Human-Computer Interaction with Integrated AI Generative Models7 Oct 2023
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis4 Jul 2023
Similar, not cited
Close in wording, no citation either way
- 2024Zero-Shot Prompting Approaches for LLM-based Graphical User Interface Generation
- 2026From Natural Language Requirements to Graphical User Interfaces: Automated Prototyping and Verification with Pretrained Language Models ↗
- 2025GUIDE: LLM-Driven GUI Generation Decomposition for Automated Prototyping
- 2026AI Prototyper: A Figma Plugin for Decomposition-Based GUI Prototyping with LLMs ↗
- 2026Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
- 2026UI Remix: Supporting UI Design Through Interactive Example Retrieval and Remixing ↗
- 2024CrowdGenUI: Aligning LLM-Based UI Generation with Crowdsourced User Preferences