# Efficient video annotation with visual interpolation and frame selection guidance

arXiv: https://arxiv.org/abs/2012.12554 · HTML: https://mlsanity.com/paper/2012.12554/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2020.

## Built on by (most cited first, 10 of 10)

- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation](https://mlsanity.com/paper/2308.15975/) (2023)
- [Beyond SOT: Tracking Multiple Generic Objects at Once](https://mlsanity.com/paper/2212.11920/) (2022)
- [Appearance-Based Refinement for Object-Centric Motion Segmentation](https://mlsanity.com/paper/2312.11463/) (2023)
- [Semi-Automatic Annotation For Visual Object Tracking](https://mlsanity.com/paper/2101.06977/) (2021)
- [Objects do not disappear: Video object detection by single-frame object location anticipation](https://mlsanity.com/paper/2308.04770/) (2023)
- [Video Annotation for Visual Tracking via Selection and Refinement](https://mlsanity.com/paper/2108.03821/) (2021)
- [Accelerated Video Annotation driven by Deep Detector and Tracker](https://mlsanity.com/paper/2302.09590/) (2023)
- [An Evaluation of Hybrid Annotation Workflows on High-Ambiguity Spatiotemporal Video Footage](https://mlsanity.com/paper/2510.21798/) (2025)
- [Collecting Consistently High Quality Object Tracks with Minimal Human Involvement by Using Self-Supervised Learning to Detect Tracker Errors](https://mlsanity.com/paper/2405.03643/) (2024)

## Abstract (arXiv)

We introduce a unified framework for generic video annotation with bounding boxes. Video annotation is a longstanding problem, as it is a tedious and time-consuming process. We tackle two important challenges of video annotation: (1) automatic temporal interpolation and extrapolation of bounding boxes provided by a human annotator on a subset of all frames, and (2) automatic selection of frames to annotate manually. Our contribution is two-fold: first, we propose a model that has both interpolating and extrapolating capabilities; second, we propose a guiding mechanism that sequentially generates suggestions for what frame to annotate next, based on the annotations made previously. We extensively evaluate our approach on several challenging datasets in simulation and demonstrate a reduction in terms of the number of manual bounding boxes drawn by 60% over linear interpolation and by 35% over an off-the-shelf tracker. Moreover, we also show 10% annotation time improvement over a state-of-the-art method for video annotation with bounding boxes [25]. Finally, we run human annotation experiments and provide extensive analysis of the results, showing that our approach reduces actual measured annotation time by 50% compared to commonly used linear interpolation.