# Sketch-based Video Object Segmentation: Benchmark and Analysis

arXiv: https://arxiv.org/abs/2311.07261 · HTML: https://mlsanity.com/paper/2311.07261/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 58 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation](https://mlsanity.com/paper/2106.05210/) (2021)
- [Language as Queries for Referring Video Object Segmentation](https://mlsanity.com/paper/2201.00487/) (2022)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)
- [Video Object Segmentation with Language Referring Expressions](https://mlsanity.com/paper/1803.08006/) (2018)
- [End-to-End Referring Video Object Segmentation with Multimodal Transformers](https://mlsanity.com/paper/2111.14821/) (2021)
- [Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion](https://mlsanity.com/paper/2103.07941/) (2021)
- [SegFlow: Joint Learning for Video Object Segmentation and Optical Flow](https://mlsanity.com/paper/1709.06750/) (2017)
- [Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks](https://mlsanity.com/paper/2001.06807/) (2020)
- [Collaborative Video Object Segmentation by Multi-Scale Foreground-Background Integration](https://mlsanity.com/paper/2010.06349/) (2020)
- [Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval](https://mlsanity.com/paper/1904.03451/) (2019)
- [Sketchformer: Transformer-based Representation for Sketched Structure](https://mlsanity.com/paper/2002.10381/) (2020)
- [Joint Inductive and Transductive Learning for Video Object Segmentation](https://mlsanity.com/paper/2108.03679/) (2021)
- [Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2002.10310/) (2020)
- [Fast and Accurate Online Video Object Segmentation via Tracking Parts](https://mlsanity.com/paper/1806.02323/) (2018)
- [Anchor Diffusion for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/1910.10895/) (2019)
- [StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval](https://mlsanity.com/paper/2103.15706/) (2021)
- [Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation](https://mlsanity.com/paper/2206.03789/) (2022)
- [FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context](https://mlsanity.com/paper/2203.02113/) (2022)
- [The 2018 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1803.00557/) (2018)

## Built on by (most cited first, 1 of 1)

- [SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models](https://mlsanity.com/paper/2508.07149/) (2025)

## Abstract (arXiv)

Reference-based video object segmentation is an emerging topic which aims to segment the corresponding target object in each video frame referred by a given reference, such as a language expression or a photo mask. However, language expressions can sometimes be vague in conveying an intended concept and ambiguous when similar objects in one frame are hard to distinguish by language. Meanwhile, photo masks are costly to annotate and less practical to provide in a real application. This paper introduces a new task of sketch-based video object segmentation, an associated benchmark, and a strong baseline. Our benchmark includes three datasets, Sketch-DAVIS16, Sketch-DAVIS17 and Sketch-YouTube-VOS, which exploit human-drawn sketches as an informative yet low-cost reference for video object segmentation. We take advantage of STCN, a popular baseline of semi-supervised VOS task, and evaluate what the most effective design for incorporating a sketch reference is. Experimental results show sketch is more effective yet annotation-efficient than other references, such as photo masks, language and scribble.