# From Lost to Found: Discover Missing UI Design Semantics through Recovering Missing Tags

arXiv: https://arxiv.org/abs/2008.06895 · HTML: https://mlsanity.com/paper/2008.06895/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2020.

## Built on by (most cited first, 4 of 4)

- [GLIB: Towards Automated Test Oracle for Graphically-Rich Applications](https://mlsanity.com/paper/2106.10507/) (2021)
- [Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Images](https://mlsanity.com/paper/2206.10352/) (2022)
- [Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug Reports](https://mlsanity.com/paper/2407.08610/) (2024)
- [Breaking Single-Tester Limits: Multi-Agent LLMs for Multi-User Feature Testing](https://mlsanity.com/paper/2506.17539/) (2025)

## Abstract (arXiv)

Design sharing sites provide UI designers with a platform to share their works and also an opportunity to get inspiration from others' designs. To facilitate management and search of millions of UI design images, many design sharing sites adopt collaborative tagging systems by distributing the work of categorization to the community. However, designers often do not know how to properly tag one design image with compact textual description, resulting in unclear, incomplete, and inconsistent tags for uploaded examples which impede retrieval, according to our empirical study and interview with four professional designers. Based on a deep neural network, we introduce a novel approach for encoding both the visual and textual information to recover the missing tags for existing UI examples so that they can be more easily found by text queries. We achieve 82.72% accuracy in the tag prediction. Through a simulation test of 5 queries, our system on average returns hundreds more results than the default Dribbble search, leading to better relatedness, diversity and satisfaction.