# ScribbleBox: Interactive Annotation Framework for Video Object Segmentation

arXiv: https://arxiv.org/abs/2008.09721 · HTML: https://mlsanity.com/paper/2008.09721/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2020.

## Built on by (most cited first, 14 of 14)

- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Full-Duplex Strategy for Video Object Segmentation](https://mlsanity.com/paper/2108.03151/) (2021)
- [Adaptive Shells for Efficient Neural Radiance Field Rendering](https://mlsanity.com/paper/2311.10091/) (2023)
- [MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2508.05630/) (2025)
- [Learning the What and How of Annotation in Video Object Segmentation](https://mlsanity.com/paper/2311.04414/) (2023)
- [VideoClick: Video Object Segmentation with a Single Click](https://mlsanity.com/paper/2101.06545/) (2021)
- [A Survey on Machine Learning Techniques for Auto Labeling of Video, Audio, and Text Data](https://mlsanity.com/paper/2109.03784/) (2021)
- [Video Annotation for Visual Tracking via Selection and Refinement](https://mlsanity.com/paper/2108.03821/) (2021)
- [Revisiting Click-based Interactive Video Object Segmentation](https://mlsanity.com/paper/2203.01784/) (2022)
- [Accelerating the creation of instance segmentation training sets through bounding box annotation](https://mlsanity.com/paper/2205.11563/) (2022)
- [ClickVOS: Click Video Object Segmentation](https://mlsanity.com/paper/2403.06130/) (2024)
- [ISAR: A Benchmark for Single- and Few-Shot Object Instance Segmentation and Re-Identification](https://mlsanity.com/paper/2311.02734/) (2023)
- [Collecting Consistently High Quality Object Tracks with Minimal Human Involvement by Using Self-Supervised Learning to Detect Tracker Errors](https://mlsanity.com/paper/2405.03643/) (2024)

## Abstract (arXiv)

Manually labeling video datasets for segmentation tasks is extremely time consuming. In this paper, we introduce ScribbleBox, a novel interactive framework for annotating object instances with masks in videos. In particular, we split annotation into two steps: annotating objects with tracked boxes, and labeling masks inside these tracks. We introduce automation and interaction in both steps. Box tracks are annotated efficiently by approximating the trajectory using a parametric curve with a small number of control points which the annotator can interactively correct. Our approach tolerates a modest amount of noise in the box placements, thus typically only a few clicks are needed to annotate tracked boxes to a sufficient accuracy. Segmentation masks are corrected via scribbles which are efficiently propagated through time. We show significant performance gains in annotation efficiency over past work. We show that our ScribbleBox approach reaches 88.92% J&F on DAVIS2017 with 9.14 clicks per box track, and 4 frames of scribble annotation.