# Compact Transformer Tracker with Correlative Masked Modeling

arXiv: https://arxiv.org/abs/2301.10938 · HTML: https://mlsanity.com/paper/2301.10938/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 49 references that are in the map.
- First posted on arXiv 26 Jan 2023.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines](https://mlsanity.com/paper/1911.06188/) (2019)
- [Acquisition of Localization Confidence for Accurate Object Detection](https://mlsanity.com/paper/1807.11590/) (2018)
- [Siam R-CNN: Visual Tracking by Re-Detection](https://mlsanity.com/paper/1911.12836/) (2019)

## Built on by (most cited first, 30 of 50)

- [Autoregressive Queries for Adaptive Tracking with Spatio-TemporalTransformers](https://mlsanity.com/paper/2403.10574/) (2024)
- [HIPTrack: Visual Tracking with Historical Prompts](https://mlsanity.com/paper/2311.02072/) (2023)
- [MixFormerV2: Efficient Fully Transformer Tracking](https://mlsanity.com/paper/2305.15896/) (2023)
- [Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking](https://mlsanity.com/paper/2412.15691/) (2024)
- [ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrieval](https://mlsanity.com/paper/2604.17898/) (2026)
- [Exploring Enhanced Contextual Information for Video-Level Object Tracking](https://mlsanity.com/paper/2412.11023/) (2024)
- [Masked Image Modeling: A Survey](https://mlsanity.com/paper/2408.06687/) (2024)
- [INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrieval](https://mlsanity.com/paper/2604.18051/) (2026)
- [HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval](https://mlsanity.com/paper/2604.18037/) (2026)
- [LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing](https://mlsanity.com/paper/2507.00029/) (2025)
- [Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrieval](https://mlsanity.com/paper/2604.19386/) (2026)
- [Autogenic Language Embedding for Coherent Point Tracking](https://mlsanity.com/paper/2407.20730/) (2024)
- [ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrieval](https://mlsanity.com/paper/2604.20358/) (2026)
- [Robust Tracking via Mamba-based Context-aware Token Learning](https://mlsanity.com/paper/2412.13611/) (2024)
- [MUST: The First Dataset and Unified Framework for Multispectral UAV Single Object Tracking](https://mlsanity.com/paper/2503.17699/) (2025)
- [TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval](https://mlsanity.com/paper/2604.21806/) (2026)
- [Cross-modulated Attention Transformer for RGBT Tracking](https://mlsanity.com/paper/2408.02222/) (2024)
- [Video Anomaly Detection with Motion and Appearance Guided Patch Diffusion Model](https://mlsanity.com/paper/2412.09026/) (2024)
- [MVP: Winning Solution to SMP Challenge 2025 Video Track](https://mlsanity.com/paper/2507.00950/) (2025)
- [Alligat0R: Pre-Training Through Co-Visibility Segmentation for Relative Camera Pose Regression](https://mlsanity.com/paper/2503.07561/) (2025)
- [Improving Visual Object Tracking through Visual Prompting](https://mlsanity.com/paper/2409.18901/) (2024)
- [Exploring Dynamic Transformer for Efficient Object Tracking](https://mlsanity.com/paper/2403.17651/) (2024)
- [DeTrack: In-model Latent Denoising Learning for Visual Object Tracking](https://mlsanity.com/paper/2501.02467/) (2025)
- [Seeing Further and Wider: Joint Spatio-Temporal Enlargement for Micro-Video Popularity Prediction](https://mlsanity.com/paper/2604.20311/) (2026)
- [Large Language Model as Token Compressor and Decompressor](https://mlsanity.com/paper/2603.25340/) (2026)
- [HotComment: A Benchmark for Evaluating Popularity of Online Comments](https://mlsanity.com/paper/2604.25614/) (2026)
- [CurEvo: Curriculum-Guided Self-Evolution for Video Understanding](https://mlsanity.com/paper/2604.26707/) (2026)
- [A Multi-Modal Transformer Network for Action Detection](https://mlsanity.com/paper/2305.19624/) (2023)
- [FARTrack: Fast Autoregressive Visual Tracking with High Performance](https://mlsanity.com/paper/2602.03214/) (2026)
- [RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.03617/) (2026)

## Abstract (arXiv)

Transformer framework has been showing superior performances in visual object tracking for its great strength in information aggregation across the template and search image with the well-known attention mechanism. Most recent advances focus on exploring attention mechanism variants for better information aggregation. We find these schemes are equivalent to or even just a subset of the basic self-attention mechanism. In this paper, we prove that the vanilla self-attention structure is sufficient for information aggregation, and structural adaption is unnecessary. The key is not the attention structure, but how to extract the discriminative feature for tracking and enhance the communication between the target and search image. Based on this finding, we adopt the basic vision transformer (ViT) architecture as our main tracker and concatenate the template and search image for feature embedding. To guide the encoder to capture the invariant feature for tracking, we attach a lightweight correlative masked decoder which reconstructs the original template and search image from the corresponding masked tokens. The correlative masked decoder serves as a plugin for the compact transform tracker and is skipped in inference. Our compact tracker uses the most simple structure which only consists of a ViT backbone and a box head, and can run at 40 fps. Extensive experiments show the proposed compact transform tracker outperforms existing approaches, including advanced attention variants, and demonstrates the sufficiency of self-attention in tracking tasks. Our method achieves state-of-the-art performance on five challenging datasets, along with the VOT2020, UAV123, LaSOT, TrackingNet, and GOT-10k benchmarks. Our project is available at https://github.com/HUSTDML/CTTrack.