# Rekall: Specifying Video Events using Compositions of Spatiotemporal Labels

arXiv: https://arxiv.org/abs/1910.02993 · HTML: https://mlsanity.com/paper/1910.02993/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2019.

## Built on by (most cited first, 14 of 14)

- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Shoring Up the Foundations: Fusing Model Embeddings and Weak Supervision](https://mlsanity.com/paper/2203.13270/) (2022)
- [Deep Learning Serves Traffic Safety Analysis: A Forward-looking Review](https://mlsanity.com/paper/2203.10939/) (2022)
- [Network-level Safety Metrics for Overall Traffic Safety Assessment: A Case Study](https://mlsanity.com/paper/2201.13229/) (2022)
- [CAST: Character labeling in Animation using Self-supervision by Tracking](https://mlsanity.com/paper/2201.07619/) (2022)
- [Uncertainty Quantification for Neurosymbolic Programs via Compositional Conformal Prediction](https://mlsanity.com/paper/2405.15912/) (2024)
- [Binary Classification with Positive Labeling Sources](https://mlsanity.com/paper/2208.01704/) (2022)
- [Supporting Experts with a Multimodal Machine-Learning-Based Tool for Human Behavior Analysis of Conversational Videos](https://mlsanity.com/paper/2402.11145/) (2024)
- [VQPy: An Object-Oriented Approach to Modern Video Analytics](https://mlsanity.com/paper/2311.01623/) (2023)
- [Convergence Behavior of an Adversarial Weak Supervision Method](https://mlsanity.com/paper/2405.16013/) (2024)
- [A Survey of Performance Optimization in Neural Network-Based Video Analytics Systems](https://mlsanity.com/paper/2105.14195/) (2021)
- [Towards Causal Physical Error Discovery in Video Analytics Systems](https://mlsanity.com/paper/2405.17686/) (2024)
- [A Survey on Efficiency Optimization Techniques for DNN-based Video Analytics: Process Systems, Algorithms, and Applications](https://mlsanity.com/paper/2507.15628/) (2025)
- [Search over the Visual World: Persistent Visual Memory, Layered Indexes, and Source-Grounded Evidence](https://mlsanity.com/paper/2608.08075/) (2026)

## Abstract (arXiv)

Many real-world video analysis applications require the ability to identify domain-specific events in video, such as interviews and commercials in TV news broadcasts, or action sequences in film. Unfortunately, pre-trained models to detect all the events of interest in video may not exist, and training new models from scratch can be costly and labor-intensive. In this paper, we explore the utility of specifying new events in video in a more traditional manner: by writing queries that compose outputs of existing, pre-trained models. To write these queries, we have developed Rekall, a library that exposes a data model and programming model for compositional video event specification. Rekall represents video annotations from different sources (object detectors, transcripts, etc.) as spatiotemporal labels associated with continuous volumes of spacetime in a video, and provides operators for composing labels into queries that model new video events. We demonstrate the use of Rekall in analyzing video from cable TV news broadcasts, films, static-camera vehicular video streams, and commercial autonomous vehicle logs. In these efforts, domain experts were able to quickly (in a few hours to a day) author queries that enabled the accurate detection of new events (on par with, and in some cases much more accurate than, learned approaches) and to rapidly retrieve video clips for human-in-the-loop tasks such as video content curation and training data curation. Finally, in a user study, novice users of Rekall were able to author queries to retrieve new events in video given just one hour of query development time.