# VideoPro: A Visual Analytics Approach for Interactive Video Programming

arXiv: https://arxiv.org/abs/2308.00401 · HTML: https://mlsanity.com/paper/2308.00401/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 85 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 30 of 43)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning](https://mlsanity.com/paper/2201.04676/) (2022)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [SF-Net: Single-Frame Supervision for Temporal Action Localization](https://mlsanity.com/paper/2003.06845/) (2020)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- [A Survey of Visual Analytics Techniques for Machine Learning](https://mlsanity.com/paper/2008.09632/) (2020)
- [MultiViz: Towards Visualizing and Understanding Multimodal Models](https://mlsanity.com/paper/2207.00056/) (2022)
- [OoDAnalyzer: Interactive Analysis of Out-of-Distribution Samples](https://mlsanity.com/paper/2002.03103/) (2020)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis](https://mlsanity.com/paper/2107.08264/) (2021)
- [Task Programming: Learning Data Efficient Behavior Representations](https://mlsanity.com/paper/2011.13917/) (2020)
- [Weakly Supervised Named Entity Tagging with Learnable Logical Rules](https://mlsanity.com/paper/2107.02282/) (2021)
- [UnweaveNet: Unweaving Activity Stories](https://mlsanity.com/paper/2112.10194/) (2021)
- [EmoCo: Visual Analysis of Emotion Coherence in Presentation Videos](https://mlsanity.com/paper/1907.12918/) (2019)
- [Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrieval](https://mlsanity.com/paper/2111.09276/) (2021)
- [Diagnosing Ensemble Few-Shot Classifiers](https://mlsanity.com/paper/2206.04372/) (2022)
- [VoiceCoach: Interactive Evidence-based Training for Voice Modulation Skills in Public Speaking](https://mlsanity.com/paper/2001.07876/) (2020)
- [Towards Visual Explainable Active Learning for Zero-Shot Classification](https://mlsanity.com/paper/2108.06730/) (2021)
- [OneLabeler: A Flexible System for Building Data Labeling Tools](https://mlsanity.com/paper/2203.14227/) (2022)
- [Explaining with Examples: Lessons Learned from Crowdsourced Introductory Description of Information Visualizations](https://mlsanity.com/paper/2112.12364/) (2021)
- [VIANA: Visual Interactive Annotation of Argumentation](https://mlsanity.com/paper/1907.12413/) (2019)
- [VASSL: A Visual Analytics Toolkit for Social Spambot Labeling](https://mlsanity.com/paper/1907.13319/) (2019)

## Built on by (most cited first, 7 of 7)

- [Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets](https://mlsanity.com/paper/2412.16839/) (2024)
- [EgoPolice: A Benchmark for Egocentric Video Understanding in High-Stakes Police Body-Worn Camera Footage](https://mlsanity.com/paper/2607.06468/) (2026)
- [POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models](https://mlsanity.com/paper/2406.03843/) (2024)
- [Multimedia and Visual Analytics in the Agentic Era](https://mlsanity.com/paper/2504.06138/) (2025)
- [RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classification](https://mlsanity.com/paper/2511.15923/) (2025)
- [Prismatic: Interactive Multi-View Cluster Analysis of Concept Stocks](https://mlsanity.com/paper/2402.08978/) (2024)
- [ProTAL: A Drag-and-Link Video Programming Framework for Temporal Action Localization](https://mlsanity.com/paper/2505.17555/) (2025)

## Abstract (arXiv)

Constructing supervised machine learning models for real-world video analysis require substantial labeled data, which is costly to acquire due to scarce domain expertise and laborious manual inspection. While data programming shows promise in generating labeled data at scale with user-defined labeling functions, the high dimensional and complex temporal information in videos poses additional challenges for effectively composing and evaluating labeling functions. In this paper, we propose VideoPro, a visual analytics approach to support flexible and scalable video data programming for model steering with reduced human effort. We first extract human-understandable events from videos using computer vision techniques and treat them as atomic components of labeling functions. We further propose a two-stage template mining algorithm that characterizes the sequential patterns of these events to serve as labeling function templates for efficient data labeling. The visual interface of VideoPro facilitates multifaceted exploration, examination, and application of the labeling templates, allowing for effective programming of video data at scale. Moreover, users can monitor the impact of programming on model performance and make informed adjustments during the iterative programming process. We demonstrate the efficiency and effectiveness of our approach with two case studies and expert interviews.