# Bridging Gulfs in UI Generation through Semantic Guidance

arXiv: https://arxiv.org/abs/2601.19171 · HTML: https://mlsanity.com/paper/2601.19171/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 60 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators](https://mlsanity.com/paper/1901.06767/) (2019)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces](https://mlsanity.com/paper/2309.14459/) (2023)
- [Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models](https://mlsanity.com/paper/2304.09337/) (2023)
- [Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation](https://mlsanity.com/paper/2310.12953/) (2023)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- [LayoutDM: Transformer-based Diffusion Model for Layout Generation](https://mlsanity.com/paper/2305.02567/) (2023)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AI](https://mlsanity.com/paper/2312.11949/) (2023)
- [DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models](https://mlsanity.com/paper/2310.03691/) (2023)
- [Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model](https://mlsanity.com/paper/2503.04084/) (2025)
- [Generating Automatic Feedback on UI Mockups with Large Language Models](https://mlsanity.com/paper/2403.13139/) (2024)
- [ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations](https://mlsanity.com/paper/2310.04869/) (2023)
- [User Experience Design Professionals' Perceptions of Generative Artificial Intelligence](https://mlsanity.com/paper/2309.15237/) (2023)
- [GenQuery: Supporting Expressive Visual Search with Generative Models](https://mlsanity.com/paper/2310.01287/) (2023)
- [UIClip: A Data-driven Model for Assessing User Interface Design](https://mlsanity.com/paper/2404.12500/) (2024)
- [Leveraging Multimodal LLM for Inspirational User Interface Search](https://mlsanity.com/paper/2501.17799/) (2025)
- [Boosting GUI Prototyping with Diffusion Models](https://mlsanity.com/paper/2306.06233/) (2023)
- [Misty: UI Prototyping Through Interactive Conceptual Blending](https://mlsanity.com/paper/2409.13900/) (2024)
- [The GenUI Study: Exploring the Design of Generative UI Tools to Support UX Practitioners and Beyond](https://mlsanity.com/paper/2501.13145/) (2025)
- [AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools](https://mlsanity.com/paper/2502.18736/) (2025)
- [Malleable Overview-Detail Interfaces](https://mlsanity.com/paper/2503.07782/) (2025)
- [GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial Network](https://mlsanity.com/paper/2203.03827/) (2022)
- [Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces](https://mlsanity.com/paper/2404.13521/) (2024)
- [Interlinking User Stories and GUI Prototyping: A Semi-Automatic LLM-based Approach](https://mlsanity.com/paper/2406.08120/) (2024)
- [Computational Approaches for App-to-App Retrieval and Design Consistency Check](https://mlsanity.com/paper/2309.10328/) (2023)

## Built on by (most cited first, 1 of 1)

- [Beyond Prompt-to-App: Accountable Translation in Teacher-Facing Agentic Authoring](https://mlsanity.com/paper/2609.04679/) (2026)

## Abstract (arXiv)

While generative AI enables high-fidelity UI generation from text prompts, users struggle to articulate design intent and evaluate or refine results-creating gulfs of execution and evaluation. To understand the information needed for UI generation, we conducted a thematic analysis of UI prompting guidelines, identifying key design semantics and discovering that they are hierarchical and interdependent. Leveraging these findings, we developed a system that enables users to specify semantics, visualize relationships, and extract how semantics are reflected in generated UIs. By making semantics serve as an intermediate representation between human intent and AI output, our system bridges both gulfs by making requirements explicit and outcomes interpretable. A comparative user study suggests that our approach enhances users' perceived control over intent expression and outcome interpretation, and facilitates more predictable iterative refinement. Our work demonstrates how explicit semantic representation enables systematic and explainable exploration of design possibilities in AI-driven UI design.