mlsanityData to 14 Oct 2026

VINS: Visual Search for Mobile User Interface Design

arXiv 2102.05216 ↗First posted 10 Feb 2021cs.HC cs.CV cs.DBcited by 37

Not on the map. Why?

Abstract (arXiv)

Searching for relative mobile user interface (UI) design examples can aid interface designers in gaining inspiration and comparing design alternatives. However, finding such design examples is challenging, especially as current search systems rely on only text-based queries and do not consider the UI structure and content into account. This paper introduces VINS, a visual search framework, that takes as input a UI image (wireframe, high-fidelity) and retrieves visually similar design examples. We first survey interface designers to better understand their example finding process. We then develop a large-scale UI dataset that provides an accurate specification of the interface's view hierarchy (i.e., all the UI components and their specific location). By utilizing this dataset, we propose an object-detection based image retrieval framework that models the UI context and hierarchical structure. The framework achieves a mean Average Precision of 76.39% for the UI detection and high performance in querying similar UI designs.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 10 of its 50 references that are in the map. First posted on arXiv 10 Feb 2021.

Builds on

10 of 50 references.

References · 10

  1. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  2. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  3. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  4. 2015Image-based Recommendations on Styles and Substitutescited by 706
  5. 2018Object Detection with Deep Learning: A Reviewcited by 528
  6. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 131
  7. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 114
  8. 2018Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Appscited by 36
  9. 2018Detecting Visual Relationships Using Box Attentioncited by 16
  10. 2015Colors -Messengers of Concepts: Visual Design Mining for Learning Color Semanticscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 6; 2024: 11; 2025: 8; 2026: 6.

37 papers.

Sort:

By area:

  1. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 120
  2. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 113
  3. 2024LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automationcited by 36
  4. 2022Towards Better Semantic Understanding of Mobile Interfacescited by 24
  5. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversationscited by 17
  6. 2024UIClip: A Data-driven Model for Assessing User Interface Designcited by 14
  7. 2021Understanding Mobile GUI: from Pixel-Words to Screen-Sentencescited by 13
  8. 2023Boosting GUI Prototyping with Diffusion Modelscited by 10
  9. 2023Mobile User Interface Element Detection Via Adaptively Prompt Tuningcited by 8
  10. 2024DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generationcited by 8
  11. 2025DesignPref: Capturing Personal Preferences in Visual Design Generationcited by 7
  12. 2024Graph4GUI: Graph Neural Networks for Representing Graphical User Interfacescited by 6
25 more
  1. 2025UI-UG: A Unified MLLM for UI Understanding and Generationcited by 5
  2. 2024AI-Assisted Causal Pathway Diagram for Human-Centered Designcited by 4
  3. 2024GUing: A Mobile GUI Search Engine using a Vision-Language Modelcited by 4
  4. 2025Vibe Coding in Product Teams: Reconfiguring AI-Assisted Workflows, Prototyping, and Collaborationcited by 4
  5. 2023Computational Approaches for App-to-App Retrieval and Design Consistency Checkcited by 3
  6. 2025PixelWeb: The First Web GUI Dataset with Pixel-Wise Labelscited by 3
  7. 2023EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal Learningcited by 2
  8. 2024GUI Element Detection Using SOTA YOLO Deep Learning Modelscited by 2
  9. 2025Explorer: Robust Collection of Interactable GUI Elementscited by 2
  10. 2025The role of large language models in UI/UX design: A systematic literature reviewcited by 2
  11. 2022UI Layers Merger: Merging UI layers via Visual Learning and Boundary Priorcited by 1
  12. 2023Pairwise GUI Dataset Construction Between Android Phones and Tabletscited by 1
  13. 2024PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLMcited by 1
  14. 2025Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthroughcited by 1
  15. 2026GhostUI: Unveiling Hidden Interactions in Mobile UIcited by 1
  16. 2026UXBench: Measuring the Actionability of LLM-Generated UX Critiquescited by 1
  17. 2022PSDoodle: Searching for App Screens via Interactive Sketchingcited by 0
  18. 2022UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attentioncited by 0
  19. 2024Computer User Interface Understanding. A New Dataset and a Learning Frameworkcited by 0
  20. 2024TA3: Testing Against Adversarial Attacks on Machine Learning Modelscited by 0
  21. 2025ControlGUI: Guiding Generative GUI Exploration through Perceptual Visual Flowcited by 0
  22. 2026ReFinE: Streamlining UI Mockup Iteration with Research Findingscited by 0
  23. 2026Efficient Personalization of Generative User Interfacescited by 0
  24. 2026SPRITE: From Static Mockups to Engine-Ready Game UIcited by 0
  25. 2026A Benchmarking Framework for Context-aware XR Interfacescited by 0