# Visual Object Tracking with Discriminative Filters and Siamese Networks: A Survey and Outlook

arXiv: https://arxiv.org/abs/2112.02838 · HTML: https://mlsanity.com/paper/2112.02838/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 107 of its 181 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 30 of 107)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)

## Built on by (most cited first, 30 of 46)

- [Generalized Relation Modeling for Transformer Tracking](https://mlsanity.com/paper/2303.16580/) (2023)
- [Single-Model and Any-Modality for Video Object Tracking](https://mlsanity.com/paper/2311.15851/) (2023)
- [Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers](https://mlsanity.com/paper/2307.04129/) (2023)
- [Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perception](https://mlsanity.com/paper/2308.16714/) (2023)
- [Separable Self and Mixed Attention Transformers for Efficient Object Tracking](https://mlsanity.com/paper/2309.03979/) (2023)
- [VastTrack: Vast Category Visual Object Tracking](https://mlsanity.com/paper/2403.03493/) (2024)
- [OSP2B: One-Stage Point-to-Box Network for 3D Siamese Tracking](https://mlsanity.com/paper/2304.11584/) (2023)
- [Towards Sequence-Level Training for Visual Tracking](https://mlsanity.com/paper/2208.05810/) (2022)
- [Towards Category Unification of 3D Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2401.11204/) (2024)
- [MambaEVT: Event Stream based Visual Object Tracking using State Space Model](https://mlsanity.com/paper/2408.10487/) (2024)
- [Siamese Object Tracking for Unmanned Aerial Vehicle: A Review and Comprehensive Analysis](https://mlsanity.com/paper/2205.04281/) (2022)
- [Improving Underwater Visual Tracking With a Large Scale Dataset and Image Enhancement](https://mlsanity.com/paper/2308.15816/) (2023)
- [Transformers in Single Object Tracking: An Experimental Survey](https://mlsanity.com/paper/2302.11867/) (2023)
- [Beyond Visual Cues: Synchronously Exploring Target-Centric Semantics for Vision-Language Tracking](https://mlsanity.com/paper/2311.17085/) (2023)
- [Improving Visual Object Tracking through Visual Prompting](https://mlsanity.com/paper/2409.18901/) (2024)
- [CompTrack: Information Bottleneck-Guided Low-Rank Dynamic Token Compression for Point Cloud Tracking](https://mlsanity.com/paper/2511.15580/) (2025)
- [FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Tracking](https://mlsanity.com/paper/2602.24133/) (2026)
- [SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Features](https://mlsanity.com/paper/2309.16987/) (2023)
- [GLT-T++: Global-Local Transformer for 3D Siamese Tracking with Ranking Loss](https://mlsanity.com/paper/2304.00242/) (2023)
- [Learning Spatial-Temporal Regularized Tensor Sparse RPCA for Background Subtraction](https://mlsanity.com/paper/2309.15576/) (2023)
- [Beyond Frame-wise Tracking: A Trajectory-based Paradigm for Efficient Point Cloud Tracking](https://mlsanity.com/paper/2509.11453/) (2025)
- [In Defense and Revival of Bayesian Filtering for Thermal Infrared Object Tracking](https://mlsanity.com/paper/2402.17098/) (2024)
- [Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm](https://mlsanity.com/paper/2509.24741/) (2025)
- [Mobile Vision Transformer-based Visual Object Tracking](https://mlsanity.com/paper/2309.05829/) (2023)
- [AViTMP: A Tracking-Specific Transformer for Single-Branch Visual Tracking](https://mlsanity.com/paper/2310.19542/) (2023)
- [CLDTracker: A Comprehensive Language Description for Visual Tracking](https://mlsanity.com/paper/2505.23704/) (2025)
- [TourPhysics: Bringing Physics to World Models for Exploration and Manipulation from a Single Image](https://mlsanity.com/paper/2609.04911/) (2026)
- [How to Track and Segment Fish without Human Annotations: A Self-Supervised Deep Learning Approach](https://mlsanity.com/paper/2208.10662/) (2022)
- [Accelerated Video Annotation driven by Deep Detector and Tracker](https://mlsanity.com/paper/2302.09590/) (2023)
- [CARPE-ID: Continuously Adaptable Re-identification for Personalized Robot Assistance](https://mlsanity.com/paper/2310.19413/) (2023)

## Abstract (arXiv)

Accurate and robust visual object tracking is one of the most challenging and fundamental computer vision problems. It entails estimating the trajectory of the target in an image sequence, given only its initial location, and segmentation, or its rough approximation in the form of a bounding box. Discriminative Correlation Filters (DCFs) and deep Siamese Networks (SNs) have emerged as dominating tracking paradigms, which have led to significant progress. Following the rapid evolution of visual object tracking in the last decade, this survey presents a systematic and thorough review of more than 90 DCFs and Siamese trackers, based on results in nine tracking benchmarks. First, we present the background theory of both the DCF and Siamese tracking core formulations. Then, we distinguish and comprehensively review the shared as well as specific open research challenges in both these tracking paradigms. Furthermore, we thoroughly analyze the performance of DCF and Siamese trackers on nine benchmarks, covering different experimental aspects of visual tracking: datasets, evaluation metrics, performance, and speed comparisons. We finish the survey by presenting recommendations and suggestions for distinguished open challenges based on our analysis.