# VINS: Visual Search for Mobile User Interface Design

arXiv: https://arxiv.org/abs/2102.05216 · HTML: https://mlsanity.com/paper/2102.05216/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 50 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 10 of 10)

- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators](https://mlsanity.com/paper/1901.06767/) (2019)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps](https://mlsanity.com/paper/1802.02312/) (2018)
- [Detecting Visual Relationships Using Box Attention](https://mlsanity.com/paper/1807.02136/) (2018)
- [Colors -Messengers of Concepts: Visual Design Mining for Learning Color Semantics](https://mlsanity.com/paper/1505.06532/) (2015)

## Built on by (most cited first, 30 of 37)

- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents](https://mlsanity.com/paper/2407.17490/) (2024)
- [LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation](https://mlsanity.com/paper/2404.16054/) (2024)
- [Towards Better Semantic Understanding of Mobile Interfaces](https://mlsanity.com/paper/2210.02663/) (2022)
- [ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations](https://mlsanity.com/paper/2310.04869/) (2023)
- [UIClip: A Data-driven Model for Assessing User Interface Design](https://mlsanity.com/paper/2404.12500/) (2024)
- [Understanding Mobile GUI: from Pixel-Words to Screen-Sentences](https://mlsanity.com/paper/2105.11941/) (2021)
- [Boosting GUI Prototyping with Diffusion Models](https://mlsanity.com/paper/2306.06233/) (2023)
- [Mobile User Interface Element Detection Via Adaptively Prompt Tuning](https://mlsanity.com/paper/2305.09699/) (2023)
- [DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation](https://mlsanity.com/paper/2410.00201/) (2024)
- [DesignPref: Capturing Personal Preferences in Visual Design Generation](https://mlsanity.com/paper/2511.20513/) (2025)
- [Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces](https://mlsanity.com/paper/2404.13521/) (2024)
- [UI-UG: A Unified MLLM for UI Understanding and Generation](https://mlsanity.com/paper/2509.24361/) (2025)
- [AI-Assisted Causal Pathway Diagram for Human-Centered Design](https://mlsanity.com/paper/2403.08111/) (2024)
- [GUing: A Mobile GUI Search Engine using a Vision-Language Model](https://mlsanity.com/paper/2405.00145/) (2024)
- [Vibe Coding in Product Teams: Reconfiguring AI-Assisted Workflows, Prototyping, and Collaboration](https://mlsanity.com/paper/2509.10652/) (2025)
- [Computational Approaches for App-to-App Retrieval and Design Consistency Check](https://mlsanity.com/paper/2309.10328/) (2023)
- [PixelWeb: The First Web GUI Dataset with Pixel-Wise Labels](https://mlsanity.com/paper/2504.16419/) (2025)
- [EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal Learning](https://mlsanity.com/paper/2309.09867/) (2023)
- [GUI Element Detection Using SOTA YOLO Deep Learning Models](https://mlsanity.com/paper/2408.03507/) (2024)
- [Explorer: Robust Collection of Interactable GUI Elements](https://mlsanity.com/paper/2504.09352/) (2025)
- [The role of large language models in UI/UX design: A systematic literature review](https://mlsanity.com/paper/2507.04469/) (2025)
- [UI Layers Merger: Merging UI layers via Visual Learning and Boundary Prior](https://mlsanity.com/paper/2206.13389/) (2022)
- [Pairwise GUI Dataset Construction Between Android Phones and Tablets](https://mlsanity.com/paper/2310.04755/) (2023)
- [PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLM](https://mlsanity.com/paper/2409.09354/) (2024)
- [Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthrough](https://mlsanity.com/paper/2512.03568/) (2025)
- [GhostUI: Unveiling Hidden Interactions in Mobile UI](https://mlsanity.com/paper/2601.19258/) (2026)
- [UXBench: Measuring the Actionability of LLM-Generated UX Critiques](https://mlsanity.com/paper/2606.16262/) (2026)
- [PSDoodle: Searching for App Screens via Interactive Sketching](https://mlsanity.com/paper/2204.01968/) (2022)
- [UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention](https://mlsanity.com/paper/2212.03440/) (2022)

## Abstract (arXiv)

Searching for relative mobile user interface (UI) design examples can aid interface designers in gaining inspiration and comparing design alternatives. However, finding such design examples is challenging, especially as current search systems rely on only text-based queries and do not consider the UI structure and content into account. This paper introduces VINS, a visual search framework, that takes as input a UI image (wireframe, high-fidelity) and retrieves visually similar design examples. We first survey interface designers to better understand their example finding process. We then develop a large-scale UI dataset that provides an accurate specification of the interface's view hierarchy (i.e., all the UI components and their specific location). By utilizing this dataset, we propose an object-detection based image retrieval framework that models the UI context and hierarchical structure. The framework achieves a mean Average Precision of 76.39% for the UI detection and high performance in querying similar UI designs.