# Transforming Model Prediction for Tracking

arXiv: https://arxiv.org/abs/2203.11192 · HTML: https://mlsanity.com/paper/2203.11192/

## Key facts

- Cited by 117 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 75 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines](https://mlsanity.com/paper/1911.06188/) (2019)
- [Siam R-CNN: Visual Tracking by Re-Detection](https://mlsanity.com/paper/1911.12836/) (2019)
- [SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking](https://mlsanity.com/paper/1911.07241/) (2019)
- [Need for Speed: A Benchmark for Higher Frame Rate Object Tracking](https://mlsanity.com/paper/1703.05884/) (2017)
- [LaSOT: A High-quality Large-scale Single Object Tracking Benchmark](https://mlsanity.com/paper/2009.03465/) (2020)

## Built on by (most cited first, 30 of 117)

- [Generalized Relation Modeling for Transformer Tracking](https://mlsanity.com/paper/2303.16580/) (2023)
- [Robust Object Modeling for Visual Tracking](https://mlsanity.com/paper/2308.05140/) (2023)
- [SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree](https://mlsanity.com/paper/2410.16268/) (2024)
- [Autoregressive Queries for Adaptive Tracking with Spatio-TemporalTransformers](https://mlsanity.com/paper/2403.10574/) (2024)
- [Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline](https://mlsanity.com/paper/2309.14611/) (2023)
- [HIPTrack: Visual Tracking with Historical Prompts](https://mlsanity.com/paper/2311.02072/) (2023)
- [A Distractor-Aware Memory for Visual Object Tracking with SAM2](https://mlsanity.com/paper/2411.17576/) (2024)
- [Joint Visual Grounding and Tracking with Natural Language Specification](https://mlsanity.com/paper/2303.12027/) (2023)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [MixFormerV2: Efficient Fully Transformer Tracking](https://mlsanity.com/paper/2305.15896/) (2023)
- [Exploring Lightweight Hierarchical Vision Transformers for Efficient Visual Tracking](https://mlsanity.com/paper/2308.06904/) (2023)
- [Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric](https://mlsanity.com/paper/2211.11010/) (2022)
- [Frame-Event Alignment and Fusion Network for High Frame Rate Tracking](https://mlsanity.com/paper/2305.15688/) (2023)
- [Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers](https://mlsanity.com/paper/2307.04129/) (2023)
- [CiteTracker: Correlating Image and Text for Visual Tracking](https://mlsanity.com/paper/2308.11322/) (2023)
- [EgoTracks: A Long-term Egocentric Visual Object Tracking Dataset](https://mlsanity.com/paper/2301.03213/) (2023)
- [Target-Aware Tracking with Long-term Context Attention](https://mlsanity.com/paper/2302.13840/) (2023)
- [Towards Real-World Visual Tracking with Temporal Contexts](https://mlsanity.com/paper/2308.10330/) (2023)
- [VastTrack: Vast Category Visual Object Tracking](https://mlsanity.com/paper/2403.03493/) (2024)
- [ZoomTrack: Target-aware Non-uniform Resizing for Efficient Visual Tracking](https://mlsanity.com/paper/2310.10071/) (2023)
- [Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers](https://mlsanity.com/paper/2407.08394/) (2024)
- [AVisT: A Benchmark for Visual Object Tracking in Adverse Visibility](https://mlsanity.com/paper/2208.06888/) (2022)
- [Mamba-FETrack: Frame-Event Tracking via State Space Model](https://mlsanity.com/paper/2404.18174/) (2024)
- [Learning Occlusion-Robust Vision Transformers for Real-Time UAV Tracking](https://mlsanity.com/paper/2504.09228/) (2025)
- [Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2304.14394/) (2023)
- [Learning Spatial-Frequency Transformer for Visual Object Tracking](https://mlsanity.com/paper/2208.08829/) (2022)
- [Instance Tracking in 3D Scenes from Egocentric Videos](https://mlsanity.com/paper/2312.04117/) (2023)
- [CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Cameras](https://mlsanity.com/paper/2401.02826/) (2024)
- [LiteTrack: Layer Pruning with Asynchronous Feature Extraction for Lightweight and Efficient Visual Tracking](https://mlsanity.com/paper/2309.09249/) (2023)
- [Self-Supervised Multi-Object Tracking with Path Consistency](https://mlsanity.com/paper/2404.05136/) (2024)

## Abstract (arXiv)

Optimization based tracking methods have been widely successful by integrating a target model prediction module, providing effective global reasoning by minimizing an objective function. While this inductive bias integrates valuable domain knowledge, it limits the expressivity of the tracking network. In this work, we therefore propose a tracker architecture employing a Transformer-based model prediction module. Transformers capture global relations with little inductive bias, allowing it to learn the prediction of more powerful target models. We further extend the model predictor to estimate a second set of weights that are applied for accurate bounding box regression. The resulting tracker relies on training and on test frame information in order to predict all weights transductively. We train the proposed tracker end-to-end and validate its performance by conducting comprehensive experiments on multiple tracking datasets. Our tracker sets a new state of the art on three benchmarks, achieving an AUC of 68.5% on the challenging LaSOT dataset.