# GRAtt-VIS: Gated Residual Attention for Auto Rectifying Video Instance Segmentation

arXiv: https://arxiv.org/abs/2305.17096 · HTML: https://mlsanity.com/paper/2305.17096/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 28 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation](https://mlsanity.com/paper/1912.04573/) (2019)
- [Video Instance Segmentation using Inter-Frame Communication Transformers](https://mlsanity.com/paper/2106.03299/) (2021)
- [In Defense of Online Models for Video Instance Segmentation](https://mlsanity.com/paper/2207.10661/) (2022)
- [VITA: Video Instance Segmentation via Object Token Association](https://mlsanity.com/paper/2206.04403/) (2022)
- [MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training](https://mlsanity.com/paper/2208.02245/) (2022)
- [Crossover Learning for Fast Online Video Instance Segmentation](https://mlsanity.com/paper/2104.05970/) (2021)
- [Video Instance Segmentation with a Propose-Reduce Paradigm](https://mlsanity.com/paper/2103.13746/) (2021)
- [A Generalized Framework for Video Instance Segmentation](https://mlsanity.com/paper/2211.08834/) (2022)
- [Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation](https://mlsanity.com/paper/2104.05606/) (2021)
- [Efficient Video Instance Segmentation via Tracklet Query and Proposal](https://mlsanity.com/paper/2203.01853/) (2022)
- [VISOLO: Grid-Based Space-Time Aggregation for Efficient Online Video Instance Segmentation](https://mlsanity.com/paper/2112.04177/) (2021)
- [InstanceFormer: An Online Video Instance Segmentation Framework](https://mlsanity.com/paper/2208.10547/) (2022)
- [Video Instance Segmentation via Multi-scale Spatio-temporal Split Attention Transformer](https://mlsanity.com/paper/2203.13253/) (2022)
- [Robust Online Video Instance Segmentation with Track Queries](https://mlsanity.com/paper/2211.09108/) (2022)
- [Tracking Instances as Queries](https://mlsanity.com/paper/2106.11963/) (2021)

## Built on by (most cited first, 9 of 9)

- [DVIS++: Improved Decoupled Framework for Universal Video Segmentation](https://mlsanity.com/paper/2312.13305/) (2023)
- [DVIS-DAQ: Improving Video Segmentation via Dynamic Anchor Queries](https://mlsanity.com/paper/2404.00086/) (2024)
- [RGNet: A Unified Clip Retrieval and Grounding Network for Long Videos](https://mlsanity.com/paper/2312.06729/) (2023)
- [Deep Learning Techniques for Video Instance Segmentation: A Survey](https://mlsanity.com/paper/2310.12393/) (2023)
- [SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Grounding](https://mlsanity.com/paper/2510.13016/) (2025)
- [PM-VIS: High-Performance Box-Supervised Video Instance Segmentation](https://mlsanity.com/paper/2404.13863/) (2024)
- [PM-VIS+: High-Performance Video Instance Segmentation without Video Annotation](https://mlsanity.com/paper/2406.19665/) (2024)
- [Local2Global query Alignment for Video Instance Segmentation](https://mlsanity.com/paper/2507.20120/) (2025)
- [SA-VIS: Sparse frame Annotations for training Video Instance Segmentation](https://mlsanity.com/paper/2606.20140/) (2026)

## Abstract (arXiv)

Recent trends in Video Instance Segmentation (VIS) have seen a growing reliance on online methods to model complex and lengthy video sequences. However, the degradation of representation and noise accumulation of the online methods, especially during occlusion and abrupt changes, pose substantial challenges. Transformer-based query propagation provides promising directions at the cost of quadratic memory attention. However, they are susceptible to the degradation of instance features due to the above-mentioned challenges and suffer from cascading effects. The detection and rectification of such errors remain largely underexplored. To this end, we introduce GRAtt-VIS, Gated Residual Attention for Video Instance Segmentation. Firstly, we leverage a Gumbel-Softmax-based gate to detect possible errors in the current frame. Next, based on the gate activation, we rectify degraded features from its past representation. Such a residual configuration alleviates the need for dedicated memory and provides a continuous stream of relevant instance features. Secondly, we propose a novel inter-instance interaction using gate activation as a mask for self-attention. This masking strategy dynamically restricts the unrepresentative instance queries in the self-attention and preserves vital information for long-term tracking. We refer to this novel combination of Gated Residual Connection and Masked Self-Attention as GRAtt block, which can easily be integrated into the existing propagation-based framework. Further, GRAtt blocks significantly reduce the attention overhead and simplify dynamic temporal modeling. GRAtt-VIS achieves state-of-the-art performance on YouTube-VIS and the highly challenging OVIS dataset, significantly improving over previous methods. Code is available at urlhttps://github.com/Tanveer81/GRAttVIS.