Bridging Gulfs in UI Generation through Semantic Guidance
Not on the map. Why?
Abstract (arXiv)
While generative AI enables high-fidelity UI generation from text prompts, users struggle to articulate design intent and evaluate or refine results-creating gulfs of execution and evaluation. To understand the information needed for UI generation, we conducted a thematic analysis of UI prompting guidelines, identifying key design semantics and discovering that they are hierarchical and interdependent. Leveraging these findings, we developed a system that enables users to specify semantics, visualize relationships, and extract how semantics are reflected in generated UIs. By making semantics serve as an intermediate representation between human intent and AI output, our system bridges both gulfs by making requirements explicit and outcomes interpretable. A comparative user study suggests that our approach enhances users' perceived control over intent expression and outcome interpretation, and facilitates more predictable iterative refinement. Our work demonstrates how explicit semantic representation enables systematic and explainable exploration of design possibilities in AI-driven UI design.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 60 references that are in the map. First posted on arXiv 27 Jan 2026.
Builds on
31 of 60 references.
References · 31
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 268
- 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 131
- 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 114
- 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
- 2023Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfacescited by 72
- 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
- 2023Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creationcited by 64
- 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 62
- 2023PromptMagician: Interactive Prompt Engineering for Text-to-Image Creationcited by 58
- 2023LayoutDM: Transformer-based Diffusion Model for Layout Generationcited by 57
- 2021Screen2Vec: Semantic Embedding of GUI Screens and GUI Componentscited by 41
19 more
- 2023CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AIcited by 34
- 2023DirectGPT: A Direct Manipulation Interface to Interact with Large Language Modelscited by 32
- 2025Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Modelcited by 26
- 2024Generating Automatic Feedback on UI Mockups with Large Language Modelscited by 23
- 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversationscited by 17
- 2023User Experience Design Professionals' Perceptions of Generative Artificial Intelligencecited by 16
- 2023GenQuery: Supporting Expressive Visual Search with Generative Modelscited by 14
- 2024UIClip: A Data-driven Model for Assessing User Interface Designcited by 14
- 2025Leveraging Multimodal LLM for Inspirational User Interface Searchcited by 13
- 2023Boosting GUI Prototyping with Diffusion Modelscited by 10
- 2024Misty: UI Prototyping Through Interactive Conceptual Blendingcited by 9
- 2025The GenUI Study: Exploring the Design of Generative UI Tools to Support UX Practitioners and Beyondcited by 9
- 2025AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Toolscited by 8
- 2025Malleable Overview-Detail Interfacescited by 7
- 2022GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Networkcited by 6
- 2024Graph4GUI: Graph Neural Networks for Representing Graphical User Interfacescited by 6
- 2024Interlinking User Stories and GUI Prototyping: A Semi-Automatic LLM-based Approachcited by 5
- 2023Computational Approaches for App-to-App Retrieval and Design Consistency Checkcited by 3
- 2025Expanding the Generative AI Design Space through Structured Prompting and Multimodal Interfacescited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.