# Video Instance Segmentation

arXiv: https://arxiv.org/abs/1905.04804 · HTML: https://mlsanity.com/paper/1905.04804/

## Key facts

- Cited by 393 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2019.

## Built on by (most cited first, 30 of 393)

- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023)
- [Language as Queries for Referring Video Object Segmentation](https://mlsanity.com/paper/2201.00487/) (2022)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [Occluded Video Instance Segmentation: A Benchmark](https://mlsanity.com/paper/2102.01558/) (2021)
- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation](https://mlsanity.com/paper/2104.04691/) (2021)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Learning to Track with Object Permanence](https://mlsanity.com/paper/2103.14258/) (2021)
- [Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models](https://mlsanity.com/paper/2311.16103/) (2023)
- [Video Instance Segmentation using Inter-Frame Communication Transformers](https://mlsanity.com/paper/2106.03299/) (2021)
- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [A Generalist Framework for Panoptic Segmentation of Images and Videos](https://mlsanity.com/paper/2210.06366/) (2022)
- [SeqFormer: Sequential Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2112.08275/) (2021)
- [In Defense of Online Models for Video Instance Segmentation](https://mlsanity.com/paper/2207.10661/) (2022)
- [VITA: Video Instance Segmentation via Object Token Association](https://mlsanity.com/paper/2206.04403/) (2022)
- [MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training](https://mlsanity.com/paper/2208.02245/) (2022)
- [Towards Grand Unification of Object Tracking](https://mlsanity.com/paper/2207.07078/) (2022)
- [High-Quality Entity Segmentation](https://mlsanity.com/paper/2211.05776/) (2022)
- [SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation](https://mlsanity.com/paper/2103.10284/) (2021)

## Abstract (arXiv)

In this paper we present a new computer vision task, named video instance segmentation. The goal of this new task is simultaneous detection, segmentation and tracking of instances in videos. In words, it is the first time that the image instance segmentation problem is extended to the video domain. To facilitate research on this new task, we propose a large-scale benchmark called YouTube-VIS, which consists of 2883 high-resolution YouTube videos, a 40-category label set and 131k high-quality instance masks. In addition, we propose a novel algorithm called MaskTrack R-CNN for this task. Our new method introduces a new tracking branch to Mask R-CNN to jointly perform the detection, segmentation and tracking tasks simultaneously. Finally, we evaluate the proposed method and several strong baselines on our new dataset. Experimental results clearly demonstrate the advantages of the proposed algorithm and reveal insight for future improvement. We believe the video instance segmentation task will motivate the community along the line of research for video understanding.