# AURORA: Navigating UI Tarpits via Automated Neural Screen Understanding

arXiv: https://arxiv.org/abs/2404.01240 · HTML: https://mlsanity.com/paper/2404.01240/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 64 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [Fill in the Blank: Context-aware Automated Text Input Generation for Mobile GUI Testing](https://mlsanity.com/paper/2212.04732/) (2022)
- [Easy over Hard: A Case Study on Deep Learning](https://mlsanity.com/paper/1703.00133/) (2017)
- [Automatic Web Testing using Curiosity-Driven Reinforcement Learning](https://mlsanity.com/paper/2103.06018/) (2021)
- [Assessing the Quality of the Steps to Reproduce in Bug Reports](https://mlsanity.com/paper/1906.07107/) (2019)
- [Automated Test Input Generation for Android: Are We There Yet?](https://mlsanity.com/paper/1503.07217/) (2015)
- [Translating Video Recordings of Mobile App Usages into Replayable Scenarios](https://mlsanity.com/paper/2005.09057/) (2020)
- [BURT: A Chatbot for Interactive Bug Reporting](https://mlsanity.com/paper/2302.06050/) (2023)
- [On Using GUI Interaction Data to Improve Text Retrieval-based Bug Localization](https://mlsanity.com/paper/2310.08083/) (2023)
- [Detecting and Summarizing GUI Changes in Evolving Mobile Apps](https://mlsanity.com/paper/1807.09440/) (2018)
- [Toward Interactive Bug Reporting for (Android App) End-Users](https://mlsanity.com/paper/2209.10062/) (2022)
- [An Empirical Investigation into the Reproduction of Bug Reports for Android Apps](https://mlsanity.com/paper/2301.01235/) (2023)
- [Automatically Discovering, Reporting and Reproducing Android Application Crashes](https://mlsanity.com/paper/1706.01130/) (2017)
- [Mining Android App Usages for Generating Actionable GUI-based Execution Scenarios](https://mlsanity.com/paper/1801.06271/) (2018)
- [VET: Identifying and Avoiding UI Exploration Tarpits](https://mlsanity.com/paper/2102.06377/) (2021)
- [V2S: A Tool for Translating Video Recordings of Mobile App Usages into Replayable Scenarios](https://mlsanity.com/paper/2103.04531/) (2021)
- [Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios](https://mlsanity.com/paper/2301.01191/) (2023)
- [Enhancing Mobile App Bug Reporting via Real-time Understanding of Reproduction Steps](https://mlsanity.com/paper/2203.12093/) (2022)

## Abstract (arXiv)

Nearly a decade of research in software engineering has focused on automating mobile app testing to help engineers in overcoming the unique challenges associated with the software platform. Much of this work has come in the form of Automated Input Generation tools (AIG tools) that dynamically explore app screens. However, such tools have repeatedly been demonstrated to achieve lower-than-expected code coverage - particularly on sophisticated proprietary apps. Prior work has illustrated that a primary cause of these coverage deficiencies is related to so-called tarpits, or complex screens that are difficult to navigate. In this paper, we take a critical step toward enabling AIG tools to effectively navigate tarpits during app exploration through a new form of automated semantic screen understanding. We introduce AURORA, a technique that learns from the visual and textual patterns that exist in mobile app UIs to automatically detect common screen designs and navigate them accordingly. The key idea of AURORA is that there are a finite number of mobile app screen designs, albeit with subtle variations, such that the general patterns of different categories of UI designs can be learned. As such, AURORA employs a multi-modal, neural screen classifier that is able to recognize the most common types of UI screen designs. After recognizing a given screen, it then applies a set of flexible and generalizable heuristics to properly navigate the screen. We evaluated AURORA both on a set of 12 apps with known tarpits from prior work, and on a new set of five of the most popular apps from the Google Play store. Our results indicate that AURORA is able to effectively navigate tarpit screens, outperforming prior approaches that avoid tarpits by 19.6% in terms of method coverage. The improvements can be attributed to AURORA's UI design classification and heuristic navigation techniques.