# MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training

arXiv: https://arxiv.org/abs/2208.02245 · HTML: https://mlsanity.com/paper/2208.02245/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 3 Aug 2022.

## Builds on (most cited first, 30 of 34)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations](https://mlsanity.com/paper/1904.05044/) (2019)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [Occluded Video Instance Segmentation: A Benchmark](https://mlsanity.com/paper/2102.01558/) (2021)
- [BoxInst: High-Performance Instance Segmentation with Box Annotations](https://mlsanity.com/paper/2012.02310/) (2020)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation](https://mlsanity.com/paper/1912.04573/) (2019)
- [STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos](https://mlsanity.com/paper/2003.08429/) (2020)
- [Video Instance Segmentation using Inter-Frame Communication Transformers](https://mlsanity.com/paper/2106.03299/) (2021)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [In Defense of Online Models for Video Instance Segmentation](https://mlsanity.com/paper/2207.10661/) (2022)
- [Crossover Learning for Fast Online Video Instance Segmentation](https://mlsanity.com/paper/2104.05970/) (2021)
- [DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision](https://mlsanity.com/paper/2105.06464/) (2021)
- [Temporally Efficient Vision Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2204.08412/) (2022)
- [Learning Video Object Segmentation from Unlabeled Videos](https://mlsanity.com/paper/2003.05020/) (2020)
- [Weakly Supervised Instance Segmentation for Videos with Temporal Mask Consistency](https://mlsanity.com/paper/2103.12886/) (2021)

## Built on by (most cited first, 30 of 94)

- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [DVIS: Decoupled Video Instance Segmentation Framework](https://mlsanity.com/paper/2306.03413/) (2023)
- [OMG-Seg: Is One Model Good Enough For All Segmentation?](https://mlsanity.com/paper/2401.10229/) (2024)
- [General Object Foundation Model for Images and Videos at Scale](https://mlsanity.com/paper/2312.09158/) (2023)
- [Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation](https://mlsanity.com/paper/2404.03645/) (2024)
- [A Generalized Framework for Video Instance Segmentation](https://mlsanity.com/paper/2211.08834/) (2022)
- [CTVIS: Consistent Training for Online Video Instance Segmentation](https://mlsanity.com/paper/2307.12616/) (2023)
- [Towards Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2304.01715/) (2023)
- [TarViS: A Unified Approach for Target-based Video Segmentation](https://mlsanity.com/paper/2301.02657/) (2023)
- [UniVS: Unified and Universal Video Segmentation with Prompts as Queries](https://mlsanity.com/paper/2402.18115/) (2024)
- [OpenVIS: Open-vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2305.16835/) (2023)
- [Video-kMaX: A Simple Unified Approach for Online and Near-Online Video Panoptic Segmentation](https://mlsanity.com/paper/2304.04694/) (2023)
- [Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation](https://mlsanity.com/paper/2303.12782/) (2023)
- [Mask-Free Video Instance Segmentation](https://mlsanity.com/paper/2303.15904/) (2023)
- [Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis](https://mlsanity.com/paper/2304.12317/) (2023)
- [OmniTracker: Unifying Object Tracking by Tracking-with-Detection](https://mlsanity.com/paper/2303.12079/) (2023)
- [DFormer: Diffusion-guided Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2306.03437/) (2023)
- [TCOVIS: Temporally Consistent Online Video Instance Segmentation](https://mlsanity.com/paper/2309.11857/) (2023)
- [Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2407.07427/) (2024)
- [UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model](https://mlsanity.com/paper/2305.12659/) (2023)
- [MDQE: Mining Discriminative Query Embeddings to Segment Occluded Instances on Challenging Videos](https://mlsanity.com/paper/2303.14395/) (2023)
- [CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2403.12455/) (2024)
- [CAVIS: Context-Aware Video Instance Segmentation](https://mlsanity.com/paper/2407.03010/) (2024)
- [Learning Streaming Video Representation via Multitask Training](https://mlsanity.com/paper/2504.20041/) (2025)
- [GRAtt-VIS: Gated Residual Attention for Auto Rectifying Video Instance Segmentation](https://mlsanity.com/paper/2305.17096/) (2023)
- [Robust Online Video Instance Segmentation with Track Queries](https://mlsanity.com/paper/2211.09108/) (2022)
- [NOVIS: A Case for End-to-End Near-Online Video Instance Segmentation](https://mlsanity.com/paper/2308.15266/) (2023)
- [EviPrompt: A Training-Free Evidential Prompt Generation Method for Segment Anything Model in Medical Images](https://mlsanity.com/paper/2311.06400/) (2023)
- [BVI-RLV: A Fully Registered Dataset for Low-Light Video Enhancement](https://mlsanity.com/paper/2407.03535/) (2024)

## Abstract (arXiv)

We propose MinVIS, a minimal video instance segmentation (VIS) framework that achieves state-of-the-art VIS performance with neither video-based architectures nor training procedures. By only training a query-based image instance segmentation model, MinVIS outperforms the previous best result on the challenging Occluded VIS dataset by over 10% AP. Since MinVIS treats frames in training videos as independent images, we can drastically sub-sample the annotated frames in training videos without any modifications. With only 1% of labeled frames, MinVIS outperforms or is comparable to fully-supervised state-of-the-art approaches on YouTube-VIS 2019/2021. Our key observation is that queries trained to be discriminative between intra-frame object instances are temporally consistent and can be used to track instances without any manually designed heuristics. MinVIS thus has the following inference pipeline: we first apply the trained query-based image instance segmentation to video frames independently. The segmented instances are then tracked by bipartite matching of the corresponding queries. This inference is done in an online fashion and does not need to process the whole video at once. MinVIS thus has the practical advantages of reducing both the labeling costs and the memory requirements, while not sacrificing the VIS performance. Code is available at: https://github.com/NVlabs/MinVIS