# TCOVIS: Temporally Consistent Online Video Instance Segmentation

arXiv: https://arxiv.org/abs/2309.11857 · HTML: https://mlsanity.com/paper/2309.11857/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 41 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [Track to Detect and Segment: An Online Multi-Object Tracker](https://mlsanity.com/paper/2103.08808/) (2021)
- [Occluded Video Instance Segmentation: A Benchmark](https://mlsanity.com/paper/2102.01558/) (2021)
- [Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation](https://mlsanity.com/paper/1912.04573/) (2019)
- [STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos](https://mlsanity.com/paper/2003.08429/) (2020)
- [Video Instance Segmentation using Inter-Frame Communication Transformers](https://mlsanity.com/paper/2106.03299/) (2021)
- [SeqFormer: Sequential Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2112.08275/) (2021)
- [In Defense of Online Models for Video Instance Segmentation](https://mlsanity.com/paper/2207.10661/) (2022)
- [VITA: Video Instance Segmentation via Object Token Association](https://mlsanity.com/paper/2206.04403/) (2022)
- [MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training](https://mlsanity.com/paper/2208.02245/) (2022)
- [SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation](https://mlsanity.com/paper/2103.10284/) (2021)
- [Crossover Learning for Fast Online Video Instance Segmentation](https://mlsanity.com/paper/2104.05970/) (2021)

## Built on by (most cited first, 18 of 18)

- [Bootstrapping Referring Multi-Object Tracking](https://mlsanity.com/paper/2406.05039/) (2024)
- [Unified Embedding Alignment for Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2407.07427/) (2024)
- [CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2403.12455/) (2024)
- [CAVIS: Context-Aware Video Instance Segmentation](https://mlsanity.com/paper/2407.03010/) (2024)
- [VISAGE: Video Instance Segmentation with Appearance-Guided Enhancement](https://mlsanity.com/paper/2312.04885/) (2023)
- [General and Task-Oriented Video Segmentation](https://mlsanity.com/paper/2407.06540/) (2024)
- [SyncVIS: Synchronized Video Instance Segmentation](https://mlsanity.com/paper/2412.00882/) (2024)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [Latest Object Memory Management for Temporally Consistent Video Instance Segmentation](https://mlsanity.com/paper/2507.19754/) (2025)
- [A Simple Video Segmenter by Tracking Objects Along Axial Trajectories](https://mlsanity.com/paper/2311.18537/) (2023)
- [Beyond Appearance: Geometric Cues for Robust Video Instance Segmentation](https://mlsanity.com/paper/2507.05948/) (2025)
- [Hierarchical Visual Prompt Learning for Continual Video Instance Segmentation](https://mlsanity.com/paper/2508.08612/) (2025)
- [LiDAR-Camera Fusion for Video Panoptic Segmentation without Video Training](https://mlsanity.com/paper/2412.20881/) (2024)
- [SeaVIS: Sound-Enhanced Association for Online Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2603.01431/) (2026)
- [Improving Weakly-supervised Video Instance Segmentation by Leveraging Spatio-temporal Consistency](https://mlsanity.com/paper/2408.16661/) (2024)
- [Local2Global query Alignment for Video Instance Segmentation](https://mlsanity.com/paper/2507.20120/) (2025)
- [No Adaptation Without Observation: Observability-Constrained Test-Time Prompt Tuning for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2606.30937/) (2026)
- [QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment](https://mlsanity.com/paper/2607.24598/) (2026)

## Abstract (arXiv)

In recent years, significant progress has been made in video instance segmentation (VIS), with many offline and online methods achieving state-of-the-art performance. While offline methods have the advantage of producing temporally consistent predictions, they are not suitable for real-time scenarios. Conversely, online methods are more practical, but maintaining temporal consistency remains a challenging task. In this paper, we propose a novel online method for video instance segmentation, called TCOVIS, which fully exploits the temporal information in a video clip. The core of our method consists of a global instance assignment strategy and a spatio-temporal enhancement module, which improve the temporal consistency of the features from two aspects. Specifically, we perform global optimal matching between the predictions and ground truth across the whole video clip, and supervise the model with the global optimal objective. We also capture the spatial feature and aggregate it with the semantic feature between frames, thus realizing the spatio-temporal enhancement. We evaluate our method on four widely adopted VIS benchmarks, namely YouTube-VIS 2019/2021/2022 and OVIS, and achieve state-of-the-art performance on all benchmarks without bells-and-whistles. For instance, on YouTube-VIS 2021, TCOVIS achieves 49.5 AP and 61.3 AP with ResNet-50 and Swin-L backbones, respectively. Code is available at https://github.com/jun-long-li/TCOVIS.