# Robust Visual Tracking by Segmentation

arXiv: https://arxiv.org/abs/2203.11191 · HTML: https://mlsanity.com/paper/2203.11191/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 63 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 30 of 45)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)
- [Siam R-CNN: Visual Tracking by Re-Detection](https://mlsanity.com/paper/1911.12836/) (2019)
- [Learning Video Object Segmentation from Static Images](https://mlsanity.com/paper/1612.02646/) (2016)
- [Need for Speed: A Benchmark for Higher Frame Rate Object Tracking](https://mlsanity.com/paper/1703.05884/) (2017)
- [Learning Target Candidate Association to Keep Track of What Not to Track](https://mlsanity.com/paper/2103.16556/) (2021)
- [Know Your Surroundings: Exploiting Scene Information for Object Tracking](https://mlsanity.com/paper/2003.11014/) (2020)
- [Online Adaptation of Convolutional Neural Networks for Video Object Segmentation](https://mlsanity.com/paper/1706.09364/) (2017)
- [Deformable Siamese Attention Networks for Visual Object Tracking](https://mlsanity.com/paper/2004.06711/) (2020)
- [Video Object Segmentation Without Temporal Information](https://mlsanity.com/paper/1709.06031/) (2017)
- [STMTrack: Template-free Visual Tracking with Space-time Memory Networks](https://mlsanity.com/paper/2104.00324/) (2021)

## Built on by (most cited first, 30 of 38)

- [Generalized Relation Modeling for Transformer Tracking](https://mlsanity.com/paper/2303.16580/) (2023)
- [Robust Object Modeling for Visual Tracking](https://mlsanity.com/paper/2308.05140/) (2023)
- [Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric](https://mlsanity.com/paper/2211.11010/) (2022)
- [Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers](https://mlsanity.com/paper/2307.04129/) (2023)
- [VastTrack: Vast Category Visual Object Tracking](https://mlsanity.com/paper/2403.03493/) (2024)
- [Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers](https://mlsanity.com/paper/2407.08394/) (2024)
- [AVisT: A Benchmark for Visual Object Tracking in Adverse Visibility](https://mlsanity.com/paper/2208.06888/) (2022)
- [DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Tasks](https://mlsanity.com/paper/2304.00571/) (2023)
- [Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2304.14394/) (2023)
- [Reading Relevant Feature from Global Representation Memory for Visual Object Tracking](https://mlsanity.com/paper/2402.14392/) (2024)
- [Beyond SOT: Tracking Multiple Generic Objects at Once](https://mlsanity.com/paper/2212.11920/) (2022)
- [Improving Underwater Visual Tracking With a Large Scale Dataset and Image Enhancement](https://mlsanity.com/paper/2308.15816/) (2023)
- [TENet: Targetness Entanglement Incorporating with Multi-Scale Pooling and Mutually-Guided Fusion for RGB-E Object Tracking](https://mlsanity.com/paper/2405.05004/) (2024)
- [Improving Visual Object Tracking through Visual Prompting](https://mlsanity.com/paper/2409.18901/) (2024)
- [360VOTS: Visual Object Tracking and Segmentation in Omnidirectional Videos](https://mlsanity.com/paper/2404.13953/) (2024)
- [Integrating Boxes and Masks: A Multi-Object Framework for Unified Visual Tracking and Segmentation](https://mlsanity.com/paper/2308.13266/) (2023)
- [Automated Conversion of Music Videos into Lyric Videos](https://mlsanity.com/paper/2308.14922/) (2023)
- [Tracking Transforming Objects: A Benchmark](https://mlsanity.com/paper/2404.18143/) (2024)
- [SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness for Hyperspectral Object Tracking](https://mlsanity.com/paper/2403.05852/) (2024)
- [TrackPGD: Efficient Adversarial Attack using Object Binary Masks against Robust Transformer Trackers](https://mlsanity.com/paper/2407.03946/) (2024)
- [NT-VOT211: A Large-Scale Benchmark for Night-time Visual Object Tracking](https://mlsanity.com/paper/2410.20421/) (2024)
- [Leveraging the Power of Data Augmentation for Transformer-based Tracking](https://mlsanity.com/paper/2309.08264/) (2023)
- [AViTMP: A Tracking-Specific Transformer for Single-Branch Visual Tracking](https://mlsanity.com/paper/2310.19542/) (2023)
- [ZJU ReLER Submission for EPIC-KITCHEN Challenge 2023: TREK-150 Single Object Tracking](https://mlsanity.com/paper/2307.02508/) (2023)
- [Transparent Object Tracking with Enhanced Fusion Module](https://mlsanity.com/paper/2309.06701/) (2023)
- [Motion Informed Needle Segmentation in Ultrasound Images](https://mlsanity.com/paper/2312.01239/) (2023)
- [PanoSAM2: Lightweight Distortion- and Memory-aware Adaptions of SAM2 for 360 Video Object Segmentation](https://mlsanity.com/paper/2604.07901/) (2026)
- [Leveraging Text-to-Image Diffusion Models for Unsupervised Visual Object Tracking](https://mlsanity.com/paper/2605.26933/) (2026)
- [Learning Spatial Distribution of Long-Term Trackers Scores](https://mlsanity.com/paper/2308.01256/) (2023)
- [Robust Visual Tracking by Motion Analyzing](https://mlsanity.com/paper/2309.03247/) (2023)

## Abstract (arXiv)

Estimating the target extent poses a fundamental challenge in visual object tracking. Typically, trackers are box-centric and fully rely on a bounding box to define the target in the scene. In practice, objects often have complex shapes and are not aligned with the image axis. In these cases, bounding boxes do not provide an accurate description of the target and often contain a majority of background pixels. We propose a segmentation-centric tracking pipeline that not only produces a highly accurate segmentation mask, but also internally works with segmentation masks instead of bounding boxes. Thus, our tracker is able to better learn a target representation that clearly differentiates the target in the scene from background content. In order to achieve the necessary robustness for the challenging tracking scenario, we propose a separate instance localization component that is used to condition the segmentation decoder when producing the output mask. We infer a bounding box from the segmentation mask, validate our tracker on challenging tracking datasets and achieve the new state of the art on LaSOT with a success AUC score of 69.7%. Since most tracking datasets do not contain mask annotations, we cannot use them to evaluate predicted segmentation masks. Instead, we validate our segmentation quality on two popular video object segmentation datasets.