# Unified Transformer Tracker for Object Tracking

arXiv: https://arxiv.org/abs/2203.15175 · HTML: https://mlsanity.com/paper/2203.15175/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 61 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking](https://mlsanity.com/paper/2004.01888/) (2020)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)

## Built on by (most cited first, 30 of 31)

- [OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning](https://mlsanity.com/paper/2403.09634/) (2024)
- [Where is my Wallet? Modeling Object Proposal Sets for Egocentric Visual Query Localization](https://mlsanity.com/paper/2211.10528/) (2022)
- [Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2304.14394/) (2023)
- [Single-Stage Visual Query Localization in Egocentric Videos](https://mlsanity.com/paper/2306.09324/) (2023)
- [OmniTracker: Unifying Object Tracking by Tracking-with-Detection](https://mlsanity.com/paper/2303.12079/) (2023)
- [Self-Supervised Multi-Object Tracking with Path Consistency](https://mlsanity.com/paper/2404.05136/) (2024)
- [In Pursuit of Many: A Review of Modern Multiple Object Tracking Systems](https://mlsanity.com/paper/2209.04796/) (2022)
- [Reading Relevant Feature from Global Representation Memory for Visual Object Tracking](https://mlsanity.com/paper/2402.14392/) (2024)
- [Beyond SOT: Tracking Multiple Generic Objects at Once](https://mlsanity.com/paper/2212.11920/) (2022)
- [Correlational Image Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2303.12670/) (2023)
- [RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations](https://mlsanity.com/paper/2412.01826/) (2024)
- [Transformers in Single Object Tracking: An Experimental Survey](https://mlsanity.com/paper/2302.11867/) (2023)
- [Beyond Visual Cues: Synchronously Exploring Target-Centric Semantics for Vision-Language Tracking](https://mlsanity.com/paper/2311.17085/) (2023)
- [Correlation-Embedded Transformer Tracking: A Single-Branch Framework](https://mlsanity.com/paper/2401.12743/) (2024)
- [Improving Visual Object Tracking through Visual Prompting](https://mlsanity.com/paper/2409.18901/) (2024)
- [Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey](https://mlsanity.com/paper/2205.10766/) (2022)
- [Integrating Boxes and Masks: A Multi-Object Framework for Unified Visual Tracking and Segmentation](https://mlsanity.com/paper/2308.13266/) (2023)
- [Prototypical Transformer as Unified Motion Learners](https://mlsanity.com/paper/2406.01559/) (2024)
- [Leveraging the Power of Data Augmentation for Transformer-based Tracking](https://mlsanity.com/paper/2309.08264/) (2023)
- [TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization](https://mlsanity.com/paper/2505.19613/) (2025)
- [CLDTracker: A Comprehensive Language Description for Visual Tracking](https://mlsanity.com/paper/2505.23704/) (2025)
- [Multi-attention Associate Prediction Network for Visual Tracking](https://mlsanity.com/paper/2403.16395/) (2024)
- [CoFormer: Collaborating with Heterogeneous Edge Devices for Scalable Transformer Inference](https://mlsanity.com/paper/2508.20375/) (2025)
- [Tracking and Segmenting Anything in Any Modality](https://mlsanity.com/paper/2511.19475/) (2025)
- [How to Backpropagate through Hungarian in Your DETR?](https://mlsanity.com/paper/2211.14448/) (2022)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2302.02814/) (2023)
- [S3Track: Self-supervised Tracking with Soft Assignment Flow](https://mlsanity.com/paper/2305.09981/) (2023)
- [Artificial intelligence optical hardware empowers high-resolution hyperspectral video understanding at 1.2 Tb/s](https://mlsanity.com/paper/2312.10639/) (2023)
- [An Approximate Dynamic Programming Framework for Occlusion-Robust Multi-Object Tracking](https://mlsanity.com/paper/2405.15137/) (2024)
- [T-SiamTPN: Temporal Siamese Transformer Pyramid Networks for Robust and Efficient UAV Tracking](https://mlsanity.com/paper/2509.12913/) (2025)

## Abstract (arXiv)

As an important area in computer vision, object tracking has formed two separate communities that respectively study Single Object Tracking (SOT) and Multiple Object Tracking (MOT). However, current methods in one tracking scenario are not easily adapted to the other due to the divergent training datasets and tracking objects of both tasks. Although UniTrack citewang2021different demonstrates that a shared appearance model with multiple heads can be used to tackle individual tracking tasks, it fails to exploit the large-scale tracking datasets for training and performs poorly on single object tracking. In this work, we present the Unified Transformer Tracker (UTT) to address tracking problems in different scenarios with one paradigm. A track transformer is developed in our UTT to track the target in both SOT and MOT. The correlation between the target and tracking frame features is exploited to localize the target. We demonstrate that both SOT and MOT tasks can be solved within this framework. The model can be simultaneously end-to-end trained by alternatively optimizing the SOT and MOT objectives on the datasets of individual tasks. Extensive experiments are conducted on several benchmarks with a unified model trained on SOT and MOT datasets. Code will be available at https://github.com/Flowerfan/Trackron.