# VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation

arXiv: https://arxiv.org/abs/2112.04177 · HTML: https://mlsanity.com/paper/2112.04177/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 35 references that are in the map.
- First posted on arXiv 8 Dec 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Mask Scoring R-CNN](https://mlsanity.com/paper/1903.00241/) (2019)
- [BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation](https://mlsanity.com/paper/2001.00309/) (2020)
- [Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation](https://mlsanity.com/paper/1912.04573/) (2019)
- [Fully Convolutional Instance-aware Semantic Segmentation](https://mlsanity.com/paper/1611.07709/) (2016)
- [STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos](https://mlsanity.com/paper/2003.08429/) (2020)
- [Deep Snake for Real-Time Instance Segmentation](https://mlsanity.com/paper/2001.01629/) (2020)
- [Video Instance Segmentation using Inter-Frame Communication Transformers](https://mlsanity.com/paper/2106.03299/) (2021)
- [InstaBoost: Boosting Instance Segmentation via Probability Map Guided Copy-Pasting](https://mlsanity.com/paper/1908.07801/) (2019)
- [SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation](https://mlsanity.com/paper/2103.10284/) (2021)
- [Crossover Learning for Fast Online Video Instance Segmentation](https://mlsanity.com/paper/2104.05970/) (2021)
- [Video Instance Segmentation with a Propose-Reduce Paradigm](https://mlsanity.com/paper/2103.13746/) (2021)
- [CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation](https://mlsanity.com/paper/2012.03400/) (2020)
- [Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation](https://mlsanity.com/paper/2104.05606/) (2021)

## Built on by (most cited first, 29 of 29)

- [VITA: Video Instance Segmentation via Object Token Association](https://mlsanity.com/paper/2206.04403/) (2022)
- [DVIS: Decoupled Video Instance Segmentation Framework](https://mlsanity.com/paper/2306.03413/) (2023)
- [A Generalized Framework for Video Instance Segmentation](https://mlsanity.com/paper/2211.08834/) (2022)
- [Towards Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2304.01715/) (2023)
- [OpenVIS: Open-vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2305.16835/) (2023)
- [Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation](https://mlsanity.com/paper/2303.12782/) (2023)
- [TCOVIS: Temporally Consistent Online Video Instance Segmentation](https://mlsanity.com/paper/2309.11857/) (2023)
- [UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model](https://mlsanity.com/paper/2305.12659/) (2023)
- [InstanceFormer: An Online Video Instance Segmentation Framework](https://mlsanity.com/paper/2208.10547/) (2022)
- [CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2403.12455/) (2024)
- [CAVIS: Context-Aware Video Instance Segmentation](https://mlsanity.com/paper/2407.03010/) (2024)
- [GRAtt-VIS: Gated Residual Attention for Auto Rectifying Video Instance Segmentation](https://mlsanity.com/paper/2305.17096/) (2023)
- [NOVIS: A Case for End-to-End Near-Online Video Instance Segmentation](https://mlsanity.com/paper/2308.15266/) (2023)
- [VISAGE: Video Instance Segmentation with Appearance-Guided Enhancement](https://mlsanity.com/paper/2312.04885/) (2023)
- [DeVIS: Making Deformable Transformers Work for Video Instance Segmentation](https://mlsanity.com/paper/2207.11103/) (2022)
- [MobileInst: Video Instance Segmentation on the Mobile](https://mlsanity.com/paper/2303.17594/) (2023)
- [General and Task-Oriented Video Segmentation](https://mlsanity.com/paper/2407.06540/) (2024)
- [Offline-to-Online Knowledge Distillation for Video Instance Segmentation](https://mlsanity.com/paper/2302.07516/) (2023)
- [NSM4D: Neural Scene Model Based Online 4D Point Cloud Sequence Understanding](https://mlsanity.com/paper/2310.08326/) (2023)
- [SyncVIS: Synchronized Video Instance Segmentation](https://mlsanity.com/paper/2412.00882/) (2024)
- [RefineVIS: Video Instance Segmentation with Temporal Attention Refinement](https://mlsanity.com/paper/2306.04774/) (2023)
- [TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation](https://mlsanity.com/paper/2312.06630/) (2023)
- [A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects](https://mlsanity.com/paper/2506.13552/) (2025)
- [Hierarchical Visual Prompt Learning for Continual Video Instance Segmentation](https://mlsanity.com/paper/2508.08612/) (2025)
- [TIVE: A Toolbox for Identifying Video Instance Segmentation Errors](https://mlsanity.com/paper/2210.08856/) (2022)
- [Maximal Cliques on Multi-Frame Proposal Graph for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2301.12352/) (2023)
- [Local2Global query Alignment for Video Instance Segmentation](https://mlsanity.com/paper/2507.20120/) (2025)
- [SA-VIS: Sparse frame Annotations for training Video Instance Segmentation](https://mlsanity.com/paper/2606.20140/) (2026)
- [QueenVIS: Rethinking Image-Only Training for Video Instance Segmentation via Query Enrichment](https://mlsanity.com/paper/2607.24598/) (2026)

## Abstract (arXiv)

For online video instance segmentation (VIS), fully utilizing the information from previous frames in an efficient manner is essential for real-time applications. Most previous methods follow a two-stage approach requiring additional computations such as RPN and RoIAlign, and do not fully exploit the available information in the video for all subtasks in VIS. In this paper, we propose a novel single-stage framework for online VIS built based on the grid structured feature representation. The grid-based features allow us to employ fully convolutional networks for real-time processing, and also to easily reuse and share features within different components. We also introduce cooperatively operating modules that aggregate information from available frames, in order to enrich the features for all subtasks in VIS. Our design fully takes advantage of previous information in a grid form for all tasks in VIS in an efficient way, and we achieved the new state-of-the-art accuracy (38.6 AP and 36.9 AP) and speed (40.0 FPS) on YouTube-VIS 2019 and 2021 datasets among online VIS methods. The code is available at https://github.com/SuHoHan95/VISOLO.