# End-to-end video instance segmentation via spatial-temporal graph neural networks

arXiv: https://arxiv.org/abs/2203.03145 · HTML: https://mlsanity.com/paper/2203.03145/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [Mask Scoring R-CNN](https://mlsanity.com/paper/1903.00241/) (2019)
- [Learning a Neural Solver for Multiple Object Tracking](https://mlsanity.com/paper/1912.07515/) (2019)
- [PolarMask: Single Shot Instance Segmentation with Polar Representation](https://mlsanity.com/paper/1909.13226/) (2019)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)
- [Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1407.1808/) (2014)
- [Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation](https://mlsanity.com/paper/1912.04573/) (2019)
- [STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos](https://mlsanity.com/paper/2003.08429/) (2020)
- [GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learning](https://mlsanity.com/paper/2006.07327/) (2020)
- [Cross-domain Detection via Graph-induced Prototype Alignment](https://mlsanity.com/paper/2003.12849/) (2020)
- [SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation](https://mlsanity.com/paper/2007.14772/) (2020)
- [Efficient Video Object Segmentation via Network Modulation](https://mlsanity.com/paper/1802.01218/) (2018)
- [Graph Neural Based End-to-end Data Association Framework for Online Multiple-Object Tracking](https://mlsanity.com/paper/1907.05315/) (2019)
- [Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation](https://mlsanity.com/paper/2010.12176/) (2020)
- [Joint Object Detection and Multi-Object Tracking with Graph Neural Networks](https://mlsanity.com/paper/2006.13164/) (2020)

## Built on by (most cited first, 16 of 16)

- [Temporally Efficient Vision Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2204.08412/) (2022)
- [TCOVIS: Temporally Consistent Online Video Instance Segmentation](https://mlsanity.com/paper/2309.11857/) (2023)
- [AIParsing: Anchor-free Instance-level Human Parsing](https://mlsanity.com/paper/2207.06854/) (2022)
- [MDQE: Mining Discriminative Query Embeddings to Segment Occluded Instances on Challenging Videos](https://mlsanity.com/paper/2303.14395/) (2023)
- [Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning](https://mlsanity.com/paper/2603.22758/) (2026)
- [Free-Form Composition Networks for Egocentric Action Recognition](https://mlsanity.com/paper/2307.06527/) (2023)
- [Selective Synergistic Learning for Video Object-Centric Learning](https://mlsanity.com/paper/2606.15527/) (2026)
- [One-stage Video Instance Segmentation: From Frame-in Frame-out to Clip-in Clip-out](https://mlsanity.com/paper/2203.06421/) (2022)
- [Maximal Cliques on Multi-Frame Proposal Graph for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2301.12352/) (2023)
- [Two-Level Temporal Relation Model for Online Video Instance Segmentation](https://mlsanity.com/paper/2210.16795/) (2022)
- [BoxVIS: Video Instance Segmentation with Box Annotations](https://mlsanity.com/paper/2303.14618/) (2023)
- [Holistically-Nested Structure-Aware Graph Neural Network for Road Extraction](https://mlsanity.com/paper/2407.02639/) (2024)
- [Spectral Graph Reasoning Network for Hyperspectral Image Classification](https://mlsanity.com/paper/2407.02647/) (2024)
- [Graph-Boosted Attentive Network for Semantic Body Parsing](https://mlsanity.com/paper/2407.05924/) (2024)
- [Generalized Event Partonomy Inference with Structured Hierarchical Predictive Learning](https://mlsanity.com/paper/2512.04219/) (2025)
- [Uncertainty-Calibrated Spatiotemporal Field Diffusion with Sparse Supervision](https://mlsanity.com/paper/2603.04431/) (2026)

## Abstract (arXiv)

Video instance segmentation is a challenging task that extends image instance segmentation to the video domain. Existing methods either rely only on single-frame information for the detection and segmentation subproblems or handle tracking as a separate post-processing step, which limit their capability to fully leverage and share useful spatial-temporal information for all the subproblems. In this paper, we propose a novel graph-neural-network (GNN) based method to handle the aforementioned limitation. Specifically, graph nodes representing instance features are used for detection and segmentation while graph edges representing instance relations are used for tracking. Both inter and intra-frame information is effectively propagated and shared via graph updates and all the subproblems (i.e. detection, segmentation and tracking) are jointly optimized in an unified framework. The performance of our method shows great improvement on the YoutubeVIS validation dataset compared to existing methods and achieves 35.2% AP with a ResNet-50 backbone, operating at 22 FPS. Code is available at http://github.com/lucaswithai/visgraph.git .