# Wireframe-Based UI Design Search Through Image Autoencoder

arXiv: https://arxiv.org/abs/2103.07085 · HTML: https://mlsanity.com/paper/2103.07085/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Mar 2021.

## Built on by (most cited first, 17 of 17)

- [ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots](https://mlsanity.com/paper/2209.08199/) (2022)
- [DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation](https://mlsanity.com/paper/2506.06251/) (2025)
- [Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial](https://mlsanity.com/paper/2305.04933/) (2023)
- [ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations](https://mlsanity.com/paper/2310.04869/) (2023)
- [Understanding Mobile GUI: from Pixel-Words to Screen-Sentences](https://mlsanity.com/paper/2105.11941/) (2021)
- [Boosting GUI Prototyping with Diffusion Models](https://mlsanity.com/paper/2306.06233/) (2023)
- [GLIB: Towards Automated Test Oracle for Graphically-Rich Applications](https://mlsanity.com/paper/2106.10507/) (2021)
- [Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis](https://mlsanity.com/paper/2204.02448/) (2022)
- [Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Images](https://mlsanity.com/paper/2206.10352/) (2022)
- [GUing: A Mobile GUI Search Engine using a Vision-Language Model](https://mlsanity.com/paper/2405.00145/) (2024)
- [UIPro: Unleashing Superior Interaction Capability For GUI Agents](https://mlsanity.com/paper/2509.17328/) (2025)
- [Computational Approaches for App-to-App Retrieval and Design Consistency Check](https://mlsanity.com/paper/2309.10328/) (2023)
- [Reimagining Application User Interface (UI) Design using Deep Learning Methods: Challenges and Opportunities](https://mlsanity.com/paper/2303.13055/) (2023)
- [Pairwise GUI Dataset Construction Between Android Phones and Tablets](https://mlsanity.com/paper/2310.04755/) (2023)
- [Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug Reports](https://mlsanity.com/paper/2407.08610/) (2024)
- [Towards an AI-Driven Video-Based American Sign Language Dictionary: Exploring Design and Usage Experience with Learners](https://mlsanity.com/paper/2504.05857/) (2025)
- [GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction](https://mlsanity.com/paper/2604.23941/) (2026)

## Abstract (arXiv)

UI design is an integral part of software development. For many developers who do not have much UI design experience, exposing them to a large database of real-application UI designs can help them quickly build up a realistic understanding of the design space for a software feature and get design inspirations from existing applications. However, existing keyword-based, image-similarity-based, and component-matching-based methods cannot reliably find relevant high-fidelity UI designs in a large database alike to the UI wireframe that the developers sketch, in face of the great variations in UI designs. In this article, we propose a deep-learning-based UI design search engine to fill in the gap. The key innovation of our search engine is to train a wireframe image autoencoder using a large database of real-application UI designs, without the need for labeling relevant UI designs. We implement our approach for Android UI design search, and conduct extensive experiments with artificially created relevant UI designs and human evaluation of UI design search results. Our experiments confirm the superior performance of our search engine over existing image-similarity or component-matching-based methods and demonstrate the usefulness of our search engine in real-world UI design tasks.