# InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation

arXiv: https://arxiv.org/abs/2301.01882 · HTML: https://mlsanity.com/paper/2301.01882/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 66 references that are in the map.
- First posted on arXiv 5 Jan 2023.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Multiple Object Tracking: A Literature Review](https://mlsanity.com/paper/1409.7618/) (2014)

## Built on by (most cited first, 13 of 13)

- [OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentation](https://mlsanity.com/paper/2307.09356/) (2023)
- [Bootstrapping Referring Multi-Object Tracking](https://mlsanity.com/paper/2406.05039/) (2024)
- [Mask-Free Video Instance Segmentation](https://mlsanity.com/paper/2303.15904/) (2023)
- [TCOVIS: Temporally Consistent Online Video Instance Segmentation](https://mlsanity.com/paper/2309.11857/) (2023)
- [MDQE: Mining Discriminative Query Embeddings to Segment Occluded Instances on Challenging Videos](https://mlsanity.com/paper/2303.14395/) (2023)
- [CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2403.12455/) (2024)
- [VISAGE: Video Instance Segmentation with Appearance-Guided Enhancement](https://mlsanity.com/paper/2312.04885/) (2023)
- [MobileInst: Video Instance Segmentation on the Mobile](https://mlsanity.com/paper/2303.17594/) (2023)
- [General and Task-Oriented Video Segmentation](https://mlsanity.com/paper/2407.06540/) (2024)
- [RefineVIS: Video Instance Segmentation with Temporal Attention Refinement](https://mlsanity.com/paper/2306.04774/) (2023)
- [Deep Learning Techniques for Video Instance Segmentation: A Survey](https://mlsanity.com/paper/2310.12393/) (2023)
- [SA-VIS: Sparse frame Annotations for training Video Instance Segmentation](https://mlsanity.com/paper/2606.20140/) (2026)
- [DynEoMT: Learning Object Dynamicity from Online Segmentation Queries](https://mlsanity.com/paper/2609.14466/) (2026)

## Abstract (arXiv)

Video instance segmentation (VIS) aims at segmenting and tracking objects in videos. Prior methods typically generate frame-level or clip-level object instances first and then associate them by either additional tracking heads or complex instance matching algorithms. This explicit instance association approach increases system complexity and fails to fully exploit temporal cues in videos. In this paper, we design a simple, fast and yet effective query-based framework for online VIS. Relying on an instance query and proposal propagation mechanism with several specially developed components, this framework can perform accurate instance association implicitly. Specifically, we generate frame-level object instances based on a set of instance query-proposal pairs propagated from previous frames. This instance query-proposal pair is learned to bind with one specific object across frames through conscientiously developed strategies. When using such a pair to predict an object instance on the current frame, not only the generated instance is automatically associated with its precursors on previous frames, but the model gets a good prior for predicting the same object. In this way, we naturally achieve implicit instance association in parallel with segmentation and elegantly take advantage of temporal clues in videos. To show the effectiveness of our method InsPro, we evaluate it on two popular VIS benchmarks, i.e., YouTube-VIS 2019 and YouTube-VIS 2021. Without bells-and-whistles, our InsPro with ResNet-50 backbone achieves 43.2 AP and 37.6 AP on these two benchmarks respectively, outperforming all other online VIS methods.