# Towards Real-World Visual Tracking with Temporal Contexts

arXiv: https://arxiv.org/abs/2308.10330 · HTML: https://mlsanity.com/paper/2308.10330/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 94 references that are in the map.
- First posted on arXiv 20 Aug 2023.

## Builds on (most cited first, 30 of 75)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020)

## Built on by (most cited first, 26 of 26)

- [Autoregressive Queries for Adaptive Tracking with Spatio-TemporalTransformers](https://mlsanity.com/paper/2403.10574/) (2024)
- [Large-Vocabulary 3D Diffusion Model with Transformer](https://mlsanity.com/paper/2309.07920/) (2023)
- [Similarity-Guided Layer-Adaptive Vision Transformer for UAV Tracking](https://mlsanity.com/paper/2503.06625/) (2025)
- [Learning Occlusion-Robust Vision Transformers for Real-Time UAV Tracking](https://mlsanity.com/paper/2504.09228/) (2025)
- [Robust Tracking via Mamba-based Context-aware Token Learning](https://mlsanity.com/paper/2412.13611/) (2024)
- [DiffTF++: 3D-aware Diffusion Transformer for Large-Vocabulary 3D Generation](https://mlsanity.com/paper/2405.08055/) (2024)
- [SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Features](https://mlsanity.com/paper/2309.16987/) (2023)
- [Prompt-Driven Temporal Domain Adaptation for Nighttime UAV Tracking](https://mlsanity.com/paper/2409.18533/) (2024)
- [Towards Real-World Aerial Vision Guidance with Categorical 6D Pose Tracker](https://mlsanity.com/paper/2401.04377/) (2024)
- [DARTer: Dynamic Adaptive Representation Tracker for Nighttime UAV Tracking](https://mlsanity.com/paper/2505.00752/) (2025)
- [Efficient Motion Prompt Learning for Robust Visual Tracking](https://mlsanity.com/paper/2505.16321/) (2025)
- [MambaNUT: Nighttime UAV Tracking via Mamba-based Adaptive Curriculum Learning](https://mlsanity.com/paper/2412.00626/) (2024)
- [Hyperspectral Mamba for Hyperspectral Object Tracking](https://mlsanity.com/paper/2509.08265/) (2025)
- [Learning Motion Blur Robust Vision Transformers for Real-Time UAV Tracking](https://mlsanity.com/paper/2407.05383/) (2024)
- [DaDiff: Domain-aware Diffusion Model for Nighttime UAV Tracking](https://mlsanity.com/paper/2410.12270/) (2024)
- [Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking](https://mlsanity.com/paper/2412.20002/) (2024)
- [Event Stream-based Visual Object Tracking: HDETrack V2 and A High-Definition Benchmark](https://mlsanity.com/paper/2502.05574/) (2025)
- [CFTrack: Enhancing Lightweight Visual Tracking through Contrastive Learning and Feature Matching](https://mlsanity.com/paper/2502.19705/) (2025)
- [CGTrack: Cascade Gating Network with Hierarchical Feature Aggregation for UAV Tracking](https://mlsanity.com/paper/2505.05936/) (2025)
- [How Far are Modern Trackers from UAV-Anti-UAV? A Million-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2512.07385/) (2025)
- [End-to-End Unmixing with Material Prompts for Hyperspectral Object Tracking](https://mlsanity.com/paper/2605.20569/) (2026)
- [Dilated Strip Attention Network for Image Restoration](https://mlsanity.com/paper/2407.18613/) (2024)
- [T-SiamTPN: Temporal Siamese Transformer Pyramid Networks for Robust and Efficient UAV Tracking](https://mlsanity.com/paper/2509.12913/) (2025)
- [MATrack: Efficient Multiscale Adaptive Tracker for Real-Time Nighttime UAV Operations](https://mlsanity.com/paper/2510.21586/) (2025)
- [Dual Prompt-Driven Feature Encoding for Nighttime UAV Tracking](https://mlsanity.com/paper/2603.19628/) (2026)
- [Dual-branch Distilled Transformer for Efficient Asymmetric UAV Tracking](https://mlsanity.com/paper/2605.28018/) (2026)

## Abstract (arXiv)

Visual tracking has made significant improvements in the past few decades. Most existing state-of-the-art trackers 1) merely aim for performance in ideal conditions while overlooking the real-world conditions; 2) adopt the tracking-by-detection paradigm, neglecting rich temporal contexts; 3) only integrate the temporal information into the template, where temporal contexts among consecutive frames are far from being fully utilized. To handle those problems, we propose a two-level framework (TCTrack) that can exploit temporal contexts efficiently. Based on it, we propose a stronger version for real-world visual tracking, i.e., TCTrack++. It boils down to two levels: features and similarity maps. Specifically, for feature extraction, we propose an attention-based temporally adaptive convolution to enhance the spatial features using temporal information, which is achieved by dynamically calibrating the convolution weights. For similarity map refinement, we introduce an adaptive temporal transformer to encode the temporal knowledge efficiently and decode it for the accurate refinement of the similarity map. To further improve the performance, we additionally introduce a curriculum learning strategy. Also, we adopt online evaluation to measure performance in real-world conditions. Exhaustive experiments on 8 wellknown benchmarks demonstrate the superiority of TCTrack++. Real-world tests directly verify that TCTrack++ can be readily used in real-world applications.