# Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

arXiv: https://arxiv.org/abs/1802.02312 · HTML: https://mlsanity.com/paper/1802.02312/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2018.

## Built on by (most cited first, 30 of 36)

- [Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping](https://mlsanity.com/paper/2411.03292/) (2024)
- [DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation](https://mlsanity.com/paper/2506.06251/) (2025)
- [VINS: Visual Search for Mobile User Interface Design](https://mlsanity.com/paper/2102.05216/) (2021)
- [Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach](https://mlsanity.com/paper/2406.16386/) (2024)
- [GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks](https://mlsanity.com/paper/2101.09978/) (2021)
- [ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations](https://mlsanity.com/paper/2310.04869/) (2023)
- [Understanding Mobile GUI: from Pixel-Words to Screen-Sentences](https://mlsanity.com/paper/2105.11941/) (2021)
- [WinClick: GUI Grounding with Multimodal Large Language Models](https://mlsanity.com/paper/2503.04730/) (2025)
- [Lexi: Self-Supervised Learning of the UI Language](https://mlsanity.com/paper/2301.10165/) (2023)
- [GLIB: Towards Automated Test Oracle for Graphically-Rich Applications](https://mlsanity.com/paper/2106.10507/) (2021)
- [AidUI: Toward Automated Recognition of Dark Patterns in User Interfaces](https://mlsanity.com/paper/2303.06782/) (2023)
- [Multimodal Icon Annotation For Mobile Applications](https://mlsanity.com/paper/2107.04452/) (2021)
- [DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation](https://mlsanity.com/paper/2410.00201/) (2024)
- [Automatically Generating Web Applications from Requirements Via Multi-Agent Test-Driven Development](https://mlsanity.com/paper/2509.25297/) (2025)
- [Magic Layouts: Structural Prior for Component Detection in User Interface Designs](https://mlsanity.com/paper/2106.07615/) (2021)
- [Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Images](https://mlsanity.com/paper/2206.10352/) (2022)
- [An Empirical Investigation into the Use of Image Captioning for Automated Software Documentation](https://mlsanity.com/paper/2301.01224/) (2023)
- [GUing: A Mobile GUI Search Engine using a Vision-Language Model](https://mlsanity.com/paper/2405.00145/) (2024)
- [Text2App: A Framework for Creating Android Apps from Text Descriptions](https://mlsanity.com/paper/2104.08301/) (2021)
- [WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learning](https://mlsanity.com/paper/2510.04097/) (2025)
- [CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design](https://mlsanity.com/paper/2511.20737/) (2025)
- [Automatic code generation from sketches of mobile applications in end-user development using Deep Learning](https://mlsanity.com/paper/2103.05704/) (2021)
- [ULDGNN: A Fragmented UI Layer Detector Based on Graph Neural Networks](https://mlsanity.com/paper/2208.06658/) (2022)
- [EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal Learning](https://mlsanity.com/paper/2309.09867/) (2023)
- [GUI Element Detection Using SOTA YOLO Deep Learning Models](https://mlsanity.com/paper/2408.03507/) (2024)
- [The role of large language models in UI/UX design: A systematic literature review](https://mlsanity.com/paper/2507.04469/) (2025)
- [CoGen: Creation of Reusable UI Components in Figma via Textual Commands](https://mlsanity.com/paper/2601.10536/) (2026)
- [AlignUI: A Method for Designing LLM-Generated UIs Aligned with User Preferences](https://mlsanity.com/paper/2601.17614/) (2026)
- [UI Layers Merger: Merging UI layers via Visual Learning and Boundary Prior](https://mlsanity.com/paper/2206.13389/) (2022)
- [Sketch2FullStack: Generating Skeleton Code of Full Stack Website and Application from Sketch using Deep Learning and Computer Vision](https://mlsanity.com/paper/2211.14607/) (2022)

## Abstract (arXiv)

It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code. This process takes place both at an application's inception and in an evolutionary context as GUI changes keep pace with evolving features. Unfortunately, this practice is challenging and time-consuming. In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks: detection, classification, and assembly. First, logical components of a GUI are detected from a mock-up artifact using either computer vision techniques or mock-up metadata. Then, software repository mining, automated dynamic analysis, and deep convolutional neural networks are utilized to accurately classify GUI-components into domain-specific types (e.g., toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates a suitable hierarchical GUI structure from which a prototype application can be automatically assembled. We implemented this approach for Android in a system called ReDraw. Our evaluation illustrates that ReDraw achieves an average GUI-component classification accuracy of 91% and assembles prototype applications that closely mirror target mock-ups in terms of visual affinity while exhibiting reasonable code structure. Interviews with industrial practitioners illustrate ReDraw's potential to improve real development workflows.