# Video2Action: Reducing Human Interactions in Action Annotation of App Tutorial Videos

arXiv: https://arxiv.org/abs/2308.03252 · HTML: https://mlsanity.com/paper/2308.03252/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2023.

## Built on by (most cited first, 8 of 8)

- [GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration](https://mlsanity.com/paper/2503.17709/) (2025)
- [Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents](https://mlsanity.com/paper/2505.12632/) (2025)
- [Agent for User: Testing Multi-User Interactive Features in TikTok](https://mlsanity.com/paper/2504.15474/) (2025)
- [ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction](https://mlsanity.com/paper/2503.20978/) (2025)
- [Breaking Single-Tester Limits: Multi-Agent LLMs for Multi-User Feature Testing](https://mlsanity.com/paper/2506.17539/) (2025)
- [An Evaluation of Hybrid Annotation Workflows on High-Ambiguity Spatiotemporal Video Footage](https://mlsanity.com/paper/2510.21798/) (2025)
- [Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals](https://mlsanity.com/paper/2602.08882/) (2026)
- [Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks](https://mlsanity.com/paper/2609.24955/) (2026)

## Abstract (arXiv)

Tutorial videos of mobile apps have become a popular and compelling way for users to learn unfamiliar app features. To make the video accessible to the users, video creators always need to annotate the actions in the video, including what actions are performed and where to tap. However, this process can be time-consuming and labor-intensive. In this paper, we introduce a lightweight approach Video2Action, to automatically generate the action scenes and predict the action locations from the video by using image-processing and deep-learning methods. The automated experiments demonstrate the good performance of Video2Action in acquiring actions from the videos, and a user study shows the usefulness of our generated action cues in assisting video creators with action annotation.