# Towards Unified Token Learning for Vision-Language Tracking

arXiv: https://arxiv.org/abs/2308.14103 · HTML: https://mlsanity.com/paper/2308.14103/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 56 references that are in the map.
- First posted on arXiv 27 Aug 2023.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [AiATrack: Attention in Attention for Transformer Visual Tracking](https://mlsanity.com/paper/2207.09603/) (2022)
- [LaSOT: A High-quality Large-scale Single Object Tracking Benchmark](https://mlsanity.com/paper/2009.03465/) (2020)
- [Learning Target Candidate Association to Keep Track of What Not to Track](https://mlsanity.com/paper/2103.16556/) (2021)
- [Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark](https://mlsanity.com/paper/2103.16746/) (2021)
- [UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling](https://mlsanity.com/paper/2111.12085/) (2021)

## Built on by (most cited first, 30 of 62)

- [iKUN: Speak to Trackers without Retraining](https://mlsanity.com/paper/2312.16245/) (2023)
- [HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval](https://mlsanity.com/paper/2512.02792/) (2025)
- [Towards Universal Modal Tracking with Online Dense Temporal Token Learning](https://mlsanity.com/paper/2507.20177/) (2025)
- [Variation-aware Vision Token Dropping for Faster Large Vision-Language Models](https://mlsanity.com/paper/2509.01552/) (2025)
- [SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overhead](https://mlsanity.com/paper/2512.00903/) (2025)
- [DTLLM-VLT: Diverse Text Generation for Visual Language Tracking Based on LLM](https://mlsanity.com/paper/2405.12139/) (2024)
- [Exploring Decoupled Spatio-Temporal Consistency Learning and Self-Prompting Evolution for Self-Supervised Tracking](https://mlsanity.com/paper/2507.21606/) (2025)
- [EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2402.18302/) (2024)
- [Efficient Fine-Tuning of Large Language Models for Automated Medical Documentation](https://mlsanity.com/paper/2409.09324/) (2024)
- [RefAV: Towards Planning-Centric Scenario Mining](https://mlsanity.com/paper/2505.20981/) (2025)
- [RobustGS: Unified Boosting of Feedforward 3D Gaussian Splatting under Low-Quality Conditions](https://mlsanity.com/paper/2508.03077/) (2025)
- [DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM](https://mlsanity.com/paper/2410.02492/) (2024)
- [Integrating Language-Derived Appearance Elements with Visual Cues in Pedestrian Detection](https://mlsanity.com/paper/2311.01025/) (2023)
- [Visual Language Tracking with Multi-modal Interaction: A Robust Benchmark](https://mlsanity.com/paper/2409.08887/) (2024)
- [How Texts Help? A Fine-grained Evaluation to Reveal the Role of Language in Vision-Language Tracking](https://mlsanity.com/paper/2411.15600/) (2024)
- [ATCTrack: Aligning Target-Context Cues with Dynamic Target States for Robust Vision-Language Tracking](https://mlsanity.com/paper/2507.19875/) (2025)
- [Beyond Visual Cues: Synchronously Exploring Target-Centric Semantics for Vision-Language Tracking](https://mlsanity.com/paper/2311.17085/) (2023)
- [ReasoningTrack: Chain-of-Thought Reasoning for Long-term Vision-Language Tracking](https://mlsanity.com/paper/2508.05221/) (2025)
- [Enhancing Vision-Language Tracking by Effectively Converting Textual Cues into Visual Cues](https://mlsanity.com/paper/2412.19648/) (2024)
- [A GEN AI Framework for Medical Note Generation](https://mlsanity.com/paper/2410.01841/) (2024)
- [MEX: Memory-efficient Approach to Referring Multi-Object Tracking](https://mlsanity.com/paper/2502.13875/) (2025)
- [Beyond Frame-wise Tracking: A Trajectory-based Paradigm for Efficient Point Cloud Tracking](https://mlsanity.com/paper/2509.11453/) (2025)
- [Conditional Prototype Rectification Prompt Learning](https://mlsanity.com/paper/2404.09872/) (2024)
- [Diffusion-Based Image Editing: An Unforeseen Adversary to Robust Invisible Watermarks](https://mlsanity.com/paper/2511.05598/) (2025)
- [VPTracker: Global Vision-Language Tracking via Visual Prompt](https://mlsanity.com/paper/2512.22799/) (2025)
- [COST: Contrastive One-Stage Transformer for Vision-Language Small Object Tracking](https://mlsanity.com/paper/2504.01321/) (2025)
- [CLDTracker: A Comprehensive Language Description for Visual Tracking](https://mlsanity.com/paper/2505.23704/) (2025)
- [ATSTrack: Enhancing Visual-Language Tracking by Aligning Temporal and Spatial Scales](https://mlsanity.com/paper/2507.00454/) (2025)
- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)
- [SSCM: A Spatial-Semantic Consistent Model for Multi-Contrast MRI Super-Resolution](https://mlsanity.com/paper/2509.18593/) (2025)

## Abstract (arXiv)

In this paper, we present a simple, flexible and effective vision-language (VL) tracking pipeline, termed MMTrack, which casts VL tracking as a token generation task. Traditional paradigms address VL tracking task indirectly with sophisticated prior designs, making them over-specialize on the features of specific architectures or mechanisms. In contrast, our proposed framework serializes language description and bounding box into a sequence of discrete tokens. In this new design paradigm, all token queries are required to perceive the desired target and directly predict spatial coordinates of the target in an auto-regressive manner. The design without other prior modules avoids multiple sub-tasks learning and hand-designed loss functions, significantly reducing the complexity of VL tracking modeling and allowing our tracker to use a simple cross-entropy loss as unified optimization objective for VL tracking task. Extensive experiments on TNL2K, LaSOT, LaSOT_rmext and OTB99-Lang benchmarks show that our approach achieves promising results, compared to other state-of-the-arts.