# Multimodal Icon Annotation For Mobile Applications

arXiv: https://arxiv.org/abs/2107.04452 · HTML: https://mlsanity.com/paper/2107.04452/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 30 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 12 of 12)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels](https://mlsanity.com/paper/2101.04893/) (2021)
- [Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps](https://mlsanity.com/paper/1802.02312/) (2018)
- [Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?](https://mlsanity.com/paper/2008.05132/) (2020)
- [Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning](https://mlsanity.com/paper/1902.11247/) (2019)
- [Resolving Referring Expressions in Images With Labeled Elements](https://mlsanity.com/paper/1810.10165/) (2018)

## Built on by (most cited first, 8 of 8)

- [ScreenAI: A Vision-Language Model for UI and Infographics Understanding](https://mlsanity.com/paper/2402.04615/) (2024)
- [ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots](https://mlsanity.com/paper/2209.08199/) (2022)
- [Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale](https://mlsanity.com/paper/2201.04100/) (2022)
- [Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis](https://mlsanity.com/paper/2204.02448/) (2022)
- [EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal Learning](https://mlsanity.com/paper/2309.09867/) (2023)
- [PeriGuru: A Peripheral Robotic Mobile App Operation Assistant based on GUI Image Understanding and Prompting with LLM](https://mlsanity.com/paper/2409.09354/) (2024)
- [UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention](https://mlsanity.com/paper/2212.03440/) (2022)
- [Multi-modal user interface control detection using cross-attention](https://mlsanity.com/paper/2604.06934/) (2026)

## Abstract (arXiv)

Annotating user interfaces (UIs) that involves localization and classification of meaningful UI elements on a screen is a critical step for many mobile applications such as screen readers and voice control of devices. Annotating object icons, such as menu, search, and arrow backward, is especially challenging due to the lack of explicit labels on screens, their similarity to pictures, and their diverse shapes. Existing studies either use view hierarchy or pixel based methods to tackle the task. Pixel based approaches are more popular as view hierarchy features on mobile platforms are often incomplete or inaccurate, however it leaves out instructional information in the view hierarchy such as resource-ids or content descriptions. We propose a novel deep learning based multi-modal approach that combines the benefits of both pixel and view hierarchy features as well as leverages the state-of-the-art object detection techniques. In order to demonstrate the utility provided, we create a high quality UI dataset by manually annotating the most commonly used 29 icons in Rico, a large scale mobile design dataset consisting of 72k UI screenshots. The experimental results indicate the effectiveness of our multi-modal approach. Our model not only outperforms a widely used object classification baseline but also pixel based object detection models. Our study sheds light on how to combine view hierarchy with pixel features for annotating UI elements.