# Owl Eyes: Spotting UI Display Issues via Visual Understanding

arXiv: https://arxiv.org/abs/2009.01417 · HTML: https://mlsanity.com/paper/2009.01417/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2020.

## Built on by (most cited first, 16 of 16)

- [DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation](https://mlsanity.com/paper/2506.06251/) (2025)
- [GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks](https://mlsanity.com/paper/2101.09978/) (2021)
- [Understanding Mobile GUI: from Pixel-Words to Screen-Sentences](https://mlsanity.com/paper/2105.11941/) (2021)
- [GLIB: Towards Automated Test Oracle for Graphically-Rich Applications](https://mlsanity.com/paper/2106.10507/) (2021)
- [AidUI: Toward Automated Recognition of Dark Patterns in User Interfaces](https://mlsanity.com/paper/2303.06782/) (2023)
- [Less Cybersickness, Please: Demystifying and Detecting Stereoscopic Visual Inconsistencies in Virtual Reality Apps](https://mlsanity.com/paper/2406.09313/) (2024)
- [ULDGNN: A Fragmented UI Layer Detector Based on Graph Neural Networks](https://mlsanity.com/paper/2208.06658/) (2022)
- [EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal Learning](https://mlsanity.com/paper/2309.09867/) (2023)
- [Intelligent Parsing: An Automated Parsing Framework for Extracting Design Semantics from E-commerce Creatives](https://mlsanity.com/paper/2312.17283/) (2023)
- [UI Layers Merger: Merging UI layers via Visual Learning and Boundary Prior](https://mlsanity.com/paper/2206.13389/) (2022)
- [Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug Reports](https://mlsanity.com/paper/2407.08610/) (2024)
- [UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention](https://mlsanity.com/paper/2212.03440/) (2022)
- [MotorEase: Automated Detection of Motor Impairment Accessibility Issues in Mobile App UIs](https://mlsanity.com/paper/2403.13690/) (2024)
- [From Exploration to Revelation: Detecting Dark Patterns in Mobile Apps](https://mlsanity.com/paper/2411.18084/) (2024)
- [GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing](https://mlsanity.com/paper/2605.29532/) (2026)
- [DragonCrawl: A Generative, Intent-Based Framework for Scalable Mobile End-to-End Testing](https://mlsanity.com/paper/2607.28750/) (2026)

## Abstract (arXiv)

Graphical User Interface (GUI) provides a visual bridge between a software application and end users, through which they can interact with each other. With the development of technology and aesthetics, the visual effects of the GUI are more and more attracting. However, such GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on different devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, OwlEye, based on deep learning for modelling visual information of the GUI screenshot. Therefore, OwlEye can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. We manually construct a large-scale labelled dataset with 4,470 GUI screenshots with UI display issues and develop a heuristics-based data augmentation method for boosting the performance of our OwlEye. The evaluation demonstrates that our OwlEye can achieve 85% precision and 84% recall in detecting UI display issues, and 90% accuracy in localizing these issues. We also evaluate OwlEye with popular Android apps on Google Play and F-droid, and successfully uncover 57 previously-undetected UI display issues with 26 of them being confirmed or fixed so far.