# Crossover Learning for Fast Online Video Instance Segmentation

arXiv: https://arxiv.org/abs/2104.05970 · HTML: https://mlsanity.com/paper/2104.05970/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 73 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 30 of 58)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://mlsanity.com/paper/2002.10857/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)

## Built on by (most cited first, 30 of 83)

- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Video Instance Segmentation using Inter-Frame Communication Transformers](https://mlsanity.com/paper/2106.03299/) (2021)
- [SeqFormer: Sequential Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2112.08275/) (2021)
- [In Defense of Online Models for Video Instance Segmentation](https://mlsanity.com/paper/2207.10661/) (2022)
- [VITA: Video Instance Segmentation via Object Token Association](https://mlsanity.com/paper/2206.04403/) (2022)
- [MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training](https://mlsanity.com/paper/2208.02245/) (2022)
- [DVIS: Decoupled Video Instance Segmentation Framework](https://mlsanity.com/paper/2306.03413/) (2023)
- [Temporally Efficient Vision Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2204.08412/) (2022)
- [A Generalized Framework for Video Instance Segmentation](https://mlsanity.com/paper/2211.08834/) (2022)
- [CTVIS: Consistent Training for Online Video Instance Segmentation](https://mlsanity.com/paper/2307.12616/) (2023)
- [Towards Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2304.01715/) (2023)
- [Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation](https://mlsanity.com/paper/2106.11958/) (2021)
- [TubeFormer-DeepLab: Video Mask Transformer](https://mlsanity.com/paper/2205.15361/) (2022)
- [Efficient Video Instance Segmentation via Tracklet Query and Proposal](https://mlsanity.com/paper/2203.01853/) (2022)
- [VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation](https://mlsanity.com/paper/2112.04177/) (2021)
- [Video Mask Transfiner for High-Quality Video Instance Segmentation](https://mlsanity.com/paper/2207.14012/) (2022)
- [OpenVIS: Open-vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2305.16835/) (2023)
- [Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation](https://mlsanity.com/paper/2303.12782/) (2023)
- [Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2310.18709/) (2023)
- [D^2Conv3D: Dynamic Dilated Convolutions for Object Segmentation in Videos](https://mlsanity.com/paper/2111.07774/) (2021)
- [Mask-Free Video Instance Segmentation](https://mlsanity.com/paper/2303.15904/) (2023)
- [OmniTracker: Unifying Object Tracking by Tracking-with-Detection](https://mlsanity.com/paper/2303.12079/) (2023)
- [TCOVIS: Temporally Consistent Online Video Instance Segmentation](https://mlsanity.com/paper/2309.11857/) (2023)
- [Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Model](https://mlsanity.com/paper/2406.19369/) (2024)
- [UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model](https://mlsanity.com/paper/2305.12659/) (2023)
- [Improving Video Instance Segmentation via Temporal Pyramid Routing](https://mlsanity.com/paper/2107.13155/) (2021)
- [Hybrid Instance-aware Temporal Fusion for Online Video Instance Segmentation](https://mlsanity.com/paper/2112.01695/) (2021)
- [STC: Spatio-Temporal Contrastive Learning for Video Instance Segmentation](https://mlsanity.com/paper/2202.03747/) (2022)
- [InstanceFormer: An Online Video Instance Segmentation Framework](https://mlsanity.com/paper/2208.10547/) (2022)
- [InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation](https://mlsanity.com/paper/2301.01882/) (2023)

## Abstract (arXiv)

Modeling temporal visual context across frames is critical for video instance segmentation (VIS) and other video understanding tasks. In this paper, we propose a fast online VIS model named CrossVIS. For temporal information modeling in VIS, we present a novel crossover learning scheme that uses the instance feature in the current frame to pixel-wisely localize the same instance in other frames. Different from previous schemes, crossover learning does not require any additional network parameters for feature enhancement. By integrating with the instance segmentation loss, crossover learning enables efficient cross-frame instance-to-pixel relation learning and brings cost-free improvement during inference. Besides, a global balanced instance embedding branch is proposed for more accurate and more stable online instance association. We conduct extensive experiments on three challenging VIS benchmarks, ie, YouTube-VIS-2019, OVIS, and YouTube-VIS-2021 to evaluate our methods. To our knowledge, CrossVIS achieves state-of-the-art performance among all online VIS methods and shows a decent trade-off between latency and accuracy. Code will be available to facilitate future research.