# Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?

arXiv: https://arxiv.org/abs/2008.05132 · HTML: https://mlsanity.com/paper/2008.05132/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2020.

## Built on by (most cited first, 29 of 29)

- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale](https://mlsanity.com/paper/2201.04100/) (2022)
- [Towards Better Semantic Understanding of Mobile Interfaces](https://mlsanity.com/paper/2210.02663/) (2022)
- [GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks](https://mlsanity.com/paper/2101.09978/) (2021)
- [ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations](https://mlsanity.com/paper/2310.04869/) (2023)
- [Understanding Mobile GUI: from Pixel-Words to Screen-Sentences](https://mlsanity.com/paper/2105.11941/) (2021)
- [From Grounding to Planning: Benchmarking Bottlenecks in Web Agents](https://mlsanity.com/paper/2409.01927/) (2024)
- [Lexi: Self-Supervised Learning of the UI Language](https://mlsanity.com/paper/2301.10165/) (2023)
- [GLIB: Towards Automated Test Oracle for Graphically-Rich Applications](https://mlsanity.com/paper/2106.10507/) (2021)
- [AidUI: Toward Automated Recognition of Dark Patterns in User Interfaces](https://mlsanity.com/paper/2303.06782/) (2023)
- [VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning](https://mlsanity.com/paper/2406.14056/) (2024)
- [TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents](https://mlsanity.com/paper/2502.08226/) (2025)
- [Multimodal Icon Annotation For Mobile Applications](https://mlsanity.com/paper/2107.04452/) (2021)
- [DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation](https://mlsanity.com/paper/2410.00201/) (2024)
- [Model-agnostic explainable artificial intelligence for object detection in image data](https://mlsanity.com/paper/2303.17249/) (2023)
- [Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Images](https://mlsanity.com/paper/2206.10352/) (2022)
- [UINav: A Practical Approach to Train On-Device Automation Agents](https://mlsanity.com/paper/2312.10170/) (2023)
- [PixelWeb: The First Web GUI Dataset with Pixel-Wise Labels](https://mlsanity.com/paper/2504.16419/) (2025)
- [EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal Learning](https://mlsanity.com/paper/2309.09867/) (2023)
- [GUI Element Detection Using SOTA YOLO Deep Learning Models](https://mlsanity.com/paper/2408.03507/) (2024)
- [UI Layers Merger: Merging UI layers via Visual Learning and Boundary Prior](https://mlsanity.com/paper/2206.13389/) (2022)
- [Reimagining Application User Interface (UI) Design using Deep Learning Methods: Challenges and Opportunities](https://mlsanity.com/paper/2303.13055/) (2023)
- [Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug Reports](https://mlsanity.com/paper/2407.08610/) (2024)
- [Grounded GUI Understanding for Vision-Based Spatial Intelligent Agent: Exemplified by Extended Reality Apps](https://mlsanity.com/paper/2409.10811/) (2024)
- [From Exploration to Revelation: Detecting Dark Patterns in Mobile Apps](https://mlsanity.com/paper/2411.18084/) (2024)
- [UISearch: Graph-Based Embeddings for Multimodal Enterprise UI Screenshots Retrieval](https://mlsanity.com/paper/2511.19380/) (2025)
- [DOne: Decoupling Structure and Rendering for High-Fidelity Design-to-Code Generation](https://mlsanity.com/paper/2604.01226/) (2026)
- [Multi-modal user interface control detection using cross-attention](https://mlsanity.com/paper/2604.06934/) (2026)

## Abstract (arXiv)

Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element detection directly borrow the mature methods from computer vision (CV) domain, including old fashioned ones that rely on traditional image processing features (e.g., canny edge, contours), and deep learning models that learn to detect from large-scale GUI data. Unfortunately, these CV methods are not originally designed with the awareness of the unique characteristics of GUIs and GUI elements and the high localization accuracy of the GUI element detection task. We conduct the first large-scale empirical study of seven representative GUI element detection methods on over 50k GUI images to understand the capabilities, limitations and effective designs of these methods. This study not only sheds the light on the technical challenges to be addressed but also informs the design of new GUI element detection methods. We accordingly design a new GUI-specific old-fashioned method for non-text GUI element detection which adopts a novel top-down coarse-to-fine strategy, and incorporate it with the mature deep learning model for GUI text detection.Our evaluation on 25,000 GUI images shows that our method significantly advances the start-of-the-art performance in GUI element detection.