# Translating Video Recordings of Mobile App Usages into Replayable Scenarios

arXiv: https://arxiv.org/abs/2005.09057 · HTML: https://mlsanity.com/paper/2005.09057/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2020.

## Built on by (most cited first, 9 of 9)

- [AidUI: Toward Automated Recognition of Dark Patterns in User Interfaces](https://mlsanity.com/paper/2303.06782/) (2023)
- [UINav: A Practical Approach to Train On-Device Automation Agents](https://mlsanity.com/paper/2312.10170/) (2023)
- [Combining Language and App UI Analysis for the Automated Assessment of Bug Reproduction Steps](https://mlsanity.com/paper/2502.04251/) (2025)
- [Explorer: Robust Collection of Interactable GUI Elements](https://mlsanity.com/paper/2504.09352/) (2025)
- [Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug Reports](https://mlsanity.com/paper/2407.08610/) (2024)
- [GUI Test Migration via Abstraction and Concretization](https://mlsanity.com/paper/2409.05028/) (2024)
- [Breaking Single-Tester Limits: Multi-Agent LLMs for Multi-User Feature Testing](https://mlsanity.com/paper/2506.17539/) (2025)
- [MotorEase: Automated Detection of Motor Impairment Accessibility Issues in Mobile App UIs](https://mlsanity.com/paper/2403.13690/) (2024)
- [AURORA: Navigating UI Tarpits via Automated Neural Screen Understanding](https://mlsanity.com/paper/2404.01240/) (2024)

## Abstract (arXiv)

Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benefit to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this paper introduces V2S, a lightweight, automated approach for translating video recordings of Android app usages into replayable scenarios. V2S is based primarily on computer vision techniques and adapts recent solutions for object detection and image classification to detect and classify user actions captured in a video, and convert these into a replayable test scenario. We performed an extensive evaluation of V2S involving 175 videos depicting 3,534 GUI-based actions collected from users exercising features and reproducing bugs from over 80 popular Android apps. Our results illustrate that V2S can accurately replay scenarios from screen recordings, and is capable of reproducing ≈ 89% of our collected videos with minimal overhead. A case study with three industrial partners illustrates the potential usefulness of V2S from the viewpoint of developers.