# Consistent Video Instance Segmentation with Inter-Frame Recurrent Attention

arXiv: https://arxiv.org/abs/2206.07011 · HTML: https://mlsanity.com/paper/2206.07011/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation](https://mlsanity.com/paper/1912.04573/) (2019)
- [STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos](https://mlsanity.com/paper/2003.08429/) (2020)
- [SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation](https://mlsanity.com/paper/2007.14772/) (2020)
- [Video Instance Segmentation using Inter-Frame Communication Transformers](https://mlsanity.com/paper/2106.03299/) (2021)
- [Crossover Learning for Fast Online Video Instance Segmentation](https://mlsanity.com/paper/2104.05970/) (2021)
- [Video Instance Segmentation with a Propose-Reduce Paradigm](https://mlsanity.com/paper/2103.13746/) (2021)
- [ISTR: End-to-End Instance Segmentation with Transformers](https://mlsanity.com/paper/2105.00637/) (2021)

## Built on by (most cited first, 2 of 2)

- [RefineVIS: Video Instance Segmentation with Temporal Attention Refinement](https://mlsanity.com/paper/2306.04774/) (2023)
- [Deep Learning Techniques for Video Instance Segmentation: A Survey](https://mlsanity.com/paper/2310.12393/) (2023)

## Abstract (arXiv)

Video instance segmentation aims at predicting object segmentation masks for each frame, as well as associating the instances across multiple frames. Recent end-to-end video instance segmentation methods are capable of performing object segmentation and instance association together in a direct parallel sequence decoding/prediction framework. Although these methods generally predict higher quality object segmentation masks, they can fail to associate instances in challenging cases because they do not explicitly model the temporal instance consistency for adjacent frames. We propose a consistent end-to-end video instance segmentation framework with Inter-Frame Recurrent Attention to model both the temporal instance consistency for adjacent frames and the global temporal context. Our extensive experiments demonstrate that the Inter-Frame Recurrent Attention significantly improves temporal instance consistency while maintaining the quality of the object segmentation masks. Our model achieves state-of-the-art accuracy on both YouTubeVIS-2019 (62.1%) and YouTubeVIS-2021 (54.7%) datasets. In addition, quantitative and qualitative results show that the proposed methods predict more temporally consistent instance segmentation masks.