# Reducing the Annotation Effort for Video Object Segmentation Datasets

arXiv: https://arxiv.org/abs/2011.01142 · HTML: https://mlsanity.com/paper/2011.01142/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2020.

## Built on by (most cited first, 15 of 15)

- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training](https://mlsanity.com/paper/2208.02245/) (2022)
- [BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Video](https://mlsanity.com/paper/2209.12118/) (2022)
- [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025)
- [A Deeper Dive Into What Deep Spatiotemporal Networks Encode: Quantifying Static vs. Dynamic Information](https://mlsanity.com/paper/2206.02846/) (2022)
- [HODOR: High-level Object Descriptors for Object Re-segmentation in Video Learned from Static Images](https://mlsanity.com/paper/2112.09131/) (2021)
- [In Pursuit of Many: A Review of Modern Multiple Object Tracking Systems](https://mlsanity.com/paper/2209.04796/) (2022)
- [Reasoning Segmentation for Images and Videos: A Survey](https://mlsanity.com/paper/2505.18816/) (2025)
- [Generating Masks from Boxes by Mining Spatio-Temporal Consistencies in Videos](https://mlsanity.com/paper/2101.02196/) (2021)
- [Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networks](https://mlsanity.com/paper/2211.01783/) (2022)
- [SPAMming Labels: Efficient Annotations for the Trackers of Tomorrow](https://mlsanity.com/paper/2404.11426/) (2024)
- [AirObject: A Temporally Evolving Graph Embedding for Object Identification](https://mlsanity.com/paper/2111.15150/) (2021)
- [Point-VOS: Pointing Up Video Object Segmentation](https://mlsanity.com/paper/2402.05917/) (2024)
- [Augmenting Moment Retrieval: Zero-Dependency Two-Stage Learning](https://mlsanity.com/paper/2510.19622/) (2025)
- [LabelBuddy: An Open Source Music and Audio Language Annotation Tagging Tool Using AI Assistance](https://mlsanity.com/paper/2603.04293/) (2026)

## Abstract (arXiv)

For further progress in video object segmentation (VOS), larger, more diverse, and more challenging datasets will be necessary. However, densely labeling every frame with pixel masks does not scale to large datasets. We use a deep convolutional network to automatically create pseudo-labels on a pixel level from much cheaper bounding box annotations and investigate how far such pseudo-labels can carry us for training state-of-the-art VOS approaches. A very encouraging result of our study is that adding a manually annotated mask in only a single video frame for each object is sufficient to generate pseudo-labels which can be used to train a VOS method to reach almost the same performance level as when training with fully segmented videos. We use this workflow to create pixel pseudo-labels for the training set of the challenging tracking dataset TAO, and we manually annotate a subset of the validation set. Together, we obtain the new TAO-VOS benchmark, which we make publicly available at www.vision.rwth-aachen.de/page/taovos. While the performance of state-of-the-art methods on existing datasets starts to saturate, TAO-VOS remains very challenging for current algorithms and reveals their shortcomings.