# Tracking Instances as Queries

arXiv: https://arxiv.org/abs/2106.11963 · HTML: https://mlsanity.com/paper/2106.11963/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 24 references that are in the map.
- First posted on arXiv 22 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Quasi-Dense Similarity Learning for Multiple Object Tracking](https://mlsanity.com/paper/2006.06664/) (2020)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation](https://mlsanity.com/paper/1912.04573/) (2019)
- [STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos](https://mlsanity.com/paper/2003.08429/) (2020)
- [SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation](https://mlsanity.com/paper/2007.14772/) (2020)
- [Video Instance Segmentation using Inter-Frame Communication Transformers](https://mlsanity.com/paper/2106.03299/) (2021)
- [Crossover Learning for Fast Online Video Instance Segmentation](https://mlsanity.com/paper/2104.05970/) (2021)
- [Video Instance Segmentation with a Propose-Reduce Paradigm](https://mlsanity.com/paper/2103.13746/) (2021)
- [Fast Video Object Segmentation using the Global Context Module](https://mlsanity.com/paper/2001.11243/) (2020)
- [CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation](https://mlsanity.com/paper/2012.03400/) (2020)
- [MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens](https://mlsanity.com/paper/2105.15168/) (2021)
- [Dual Semantic Fusion Network for Video Object Detection](https://mlsanity.com/paper/2009.07498/) (2020)
- [1st Place Solution for YouTubeVOS Challenge 2021:Video Instance Segmentation](https://mlsanity.com/paper/2106.06649/) (2021)
- [MSN: Efficient Online Mask Selection Network for Video Instance Segmentation](https://mlsanity.com/paper/2106.10452/) (2021)

## Built on by (most cited first, 7 of 7)

- [MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training](https://mlsanity.com/paper/2208.02245/) (2022)
- [Temporally Efficient Vision Transformer for Video Instance Segmentation](https://mlsanity.com/paper/2204.08412/) (2022)
- [Look Before You Match: Instance Understanding Matters in Video Object Segmentation](https://mlsanity.com/paper/2212.06826/) (2022)
- [GRAtt-VIS: Gated Residual Attention for Auto Rectifying Video Instance Segmentation](https://mlsanity.com/paper/2305.17096/) (2023)
- [Instance As Identity: A Generic Online Paradigm for Video Instance Segmentation](https://mlsanity.com/paper/2208.03079/) (2022)
- [What is Point Supervision Worth in Video Instance Segmentation?](https://mlsanity.com/paper/2404.01990/) (2024)
- [UVIS: Unsupervised Video Instance Segmentation](https://mlsanity.com/paper/2406.06908/) (2024)

## Abstract (arXiv)

Recently, query based deep networks catch lots of attention owing to their end-to-end pipeline and competitive results on several fundamental computer vision tasks, such as object detection, semantic segmentation, and instance segmentation. However, how to establish a query based video instance segmentation (VIS) framework with elegant architecture and strong performance remains to be settled. In this paper, we present QueryTrack (i.e., tracking instances as queries), a unified query based VIS framework fully leveraging the intrinsic one-to-one correspondence between instances and queries in QueryInst. The proposed method obtains 52.7 / 52.3 AP on YouTube-VIS-2019 / 2021 datasets, which wins the 2-nd place in the YouTube-VIS Challenge at CVPR 2021 with a single online end-to-end model, single scale testing & modest amount of training data. We also provide QueryTrack-ResNet-50 baseline results on YouTube-VIS-2021 val set as references for the VIS community.