# Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning

arXiv: https://arxiv.org/abs/2003.00380 · HTML: https://mlsanity.com/paper/2003.00380/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2020.

## Built on by (most cited first, 25 of 25)

- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [UIBert: Learning Generic Multimodal Representations for UI Understanding](https://mlsanity.com/paper/2107.13731/) (2021)
- [ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots](https://mlsanity.com/paper/2209.08199/) (2022)
- [Towards Better Semantic Understanding of Mobile Interfaces](https://mlsanity.com/paper/2210.02663/) (2022)
- [GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks](https://mlsanity.com/paper/2101.09978/) (2021)
- [Understanding Mobile GUI: from Pixel-Words to Screen-Sentences](https://mlsanity.com/paper/2105.11941/) (2021)
- [Lexi: Self-Supervised Learning of the UI Language](https://mlsanity.com/paper/2301.10165/) (2023)
- [GLIB: Towards Automated Test Oracle for Graphically-Rich Applications](https://mlsanity.com/paper/2106.10507/) (2021)
- [AidUI: Toward Automated Recognition of Dark Patterns in User Interfaces](https://mlsanity.com/paper/2303.06782/) (2023)
- [VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning](https://mlsanity.com/paper/2406.14056/) (2024)
- [TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents](https://mlsanity.com/paper/2502.08226/) (2025)
- [Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Images](https://mlsanity.com/paper/2206.10352/) (2022)
- [Automated Query Reformulation for Efficient Search based on Query Logs From Stack Overflow](https://mlsanity.com/paper/2102.00826/) (2021)
- [An Empirical Investigation into the Use of Image Captioning for Automated Software Documentation](https://mlsanity.com/paper/2301.01224/) (2023)
- [AccessGuru: Leveraging LLMs to Detect and Correct Web Accessibility Violations in HTML Code](https://mlsanity.com/paper/2507.19549/) (2025)
- [The Impact of Element Ordering on LM Agent Performance](https://mlsanity.com/paper/2409.12089/) (2024)
- [ULDGNN: A Fragmented UI Layer Detector Based on Graph Neural Networks](https://mlsanity.com/paper/2208.06658/) (2022)
- [UI Layers Merger: Merging UI layers via Visual Learning and Boundary Prior](https://mlsanity.com/paper/2206.13389/) (2022)
- [Pairwise GUI Dataset Construction Between Android Phones and Tablets](https://mlsanity.com/paper/2310.04755/) (2023)
- [PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLM](https://mlsanity.com/paper/2409.09354/) (2024)
- [UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention](https://mlsanity.com/paper/2212.03440/) (2022)
- [MotorEase: Automated Detection of Motor Impairment Accessibility Issues in Mobile App UIs](https://mlsanity.com/paper/2403.13690/) (2024)
- [From Exploration to Revelation: Detecting Dark Patterns in Mobile Apps](https://mlsanity.com/paper/2411.18084/) (2024)
- [SemLink: A Semantic-Aware Automated Test Oracle for Hyperlink Verification using Siamese Sentence-BERT](https://mlsanity.com/paper/2604.05711/) (2026)

## Abstract (arXiv)

According to the World Health Organization(WHO), it is estimated that approximately 1.3 billion people live with some forms of vision impairment globally, of whom 36 million are blind. Due to their disability, engaging these minority into the society is a challenging problem. The recent rise of smart mobile phones provides a new solution by enabling blind users' convenient access to the information and service for understanding the world. Users with vision impairment can adopt the screen reader embedded in the mobile operating systems to read the content of each screen within the app, and use gestures to interact with the phone. However, the prerequisite of using screen readers is that developers have to add natural-language labels to the image-based components when they are developing the app. Unfortunately, more than 77% apps have issues of missing labels, according to our analysis of 10,408 Android apps. Most of these issues are caused by developers' lack of awareness and knowledge in considering the minority. And even if developers want to add the labels to UI components, they may not come up with concise and clear description as most of them are of no visual issues. To overcome these challenges, we develop a deep-learning based model, called LabelDroid, to automatically predict the labels of image-based buttons by learning from large-scale commercial apps in Google Play. The experimental results show that our model can make accurate predictions and the generated labels are of higher quality than that from real Android developers.