# DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM

arXiv: https://arxiv.org/abs/2410.02492 · HTML: https://mlsanity.com/paper/2410.02492/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Osprey: Pixel Understanding with Visual Instruction Tuning](https://mlsanity.com/paper/2312.10032/) (2023)
- [LaSOT: A High-quality Large-scale Single Object Tracking Benchmark](https://mlsanity.com/paper/2009.03465/) (2020)
- [Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark](https://mlsanity.com/paper/2103.16746/) (2021)
- [A Novel Performance Evaluation Methodology for Single-Target Trackers](https://mlsanity.com/paper/1503.01313/) (2015)
- [Towards Unified Token Learning for Vision-Language Tracking](https://mlsanity.com/paper/2308.14103/) (2023)
- [Joint Visual Grounding and Tracking with Natural Language Specification](https://mlsanity.com/paper/2303.12027/) (2023)
- [Divert More Attention to Vision-Language Tracking](https://mlsanity.com/paper/2207.01076/) (2022)
- [Grounding-Tracking-Integration](https://mlsanity.com/paper/1912.06316/) (2019)
- [Unifying Visual and Vision-Language Tracking via Contrastive Learning](https://mlsanity.com/paper/2401.11228/) (2024)
- [Real-time Visual Object Tracking with Natural Language Description](https://mlsanity.com/paper/1907.11751/) (2019)
- [WebUAV-3M: A Benchmark for Unveiling the Power of Million-Scale Deep UAV Tracking](https://mlsanity.com/paper/2201.07425/) (2022)
- [VastTrack: Vast Category Visual Object Tracking](https://mlsanity.com/paper/2403.03493/) (2024)
- [DTLLM-VLT: Diverse Text Generation for Visual Language Tracking Based on LLM](https://mlsanity.com/paper/2405.12139/) (2024)
- [Context-Aware Integration of Language and Visual References for Natural Language Tracking](https://mlsanity.com/paper/2403.19975/) (2024)
- [Describe and Attend to Track: Learning Natural Language guided Structural Representation and Visual Attention for Object Tracking](https://mlsanity.com/paper/1811.10014/) (2018)
- [Global Instance Tracking: Locating Target More Like Humans](https://mlsanity.com/paper/2202.13073/) (2022)
- [Siamese Natural Language Tracker: Tracking by Natural Language Descriptions with Siamese Trackers](https://mlsanity.com/paper/1912.02048/) (2019)
- [ControlCap: Controllable Region-level Captioning](https://mlsanity.com/paper/2401.17910/) (2024)
- [BioDrone: A Bionic Drone-based Single Object Tracking Benchmark for Robust Vision](https://mlsanity.com/paper/2402.04519/) (2024)
- [SOTVerse: A User-defined Task Space of Single Object Tracking](https://mlsanity.com/paper/2204.07414/) (2022)
- [Visual Language Tracking with Multi-modal Interaction: A Robust Benchmark](https://mlsanity.com/paper/2409.08887/) (2024)

## Built on by (most cited first, 12 of 12)

- [NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generation](https://mlsanity.com/paper/2507.11245/) (2025)
- [How Texts Help? A Fine-grained Evaluation to Reveal the Role of Language in Vision-Language Tracking](https://mlsanity.com/paper/2411.15600/) (2024)
- [CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features](https://mlsanity.com/paper/2505.19434/) (2025)
- [ATCTrack: Aligning Target-Context Cues with Dynamic Target States for Robust Vision-Language Tracking](https://mlsanity.com/paper/2507.19875/) (2025)
- [CausalStep: A Benchmark for Explicit Stepwise Causal Reasoning in Videos](https://mlsanity.com/paper/2507.16878/) (2025)
- [Look Less, Reason More: Rollout-Guided Adaptive Pixel-Space Reasoning](https://mlsanity.com/paper/2510.01681/) (2025)
- [Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoning](https://mlsanity.com/paper/2510.15440/) (2025)
- [DARTer: Dynamic Adaptive Representation Tracker for Nighttime UAV Tracking](https://mlsanity.com/paper/2505.00752/) (2025)
- [COST: Contrastive One-Stage Transformer for Vision-Language Small Object Tracking](https://mlsanity.com/paper/2504.01321/) (2025)
- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)
- [MathMixup: Boosting LLM Mathematical Reasoning with Difficulty-Controllable Data Synthesis and Curriculum Learning](https://mlsanity.com/paper/2601.17006/) (2026)
- [Vision-Language Guided Hyperspectral Object Tracking via Semantics Fusion and Contextual Template Updating](https://mlsanity.com/paper/2606.09167/) (2026)

## Abstract (arXiv)

Visual language tracking (VLT) has emerged as a cutting-edge research area, harnessing linguistic data to enhance algorithms with multi-modal inputs and broadening the scope of traditional single object tracking (SOT) to encompass video understanding applications. Despite this, most VLT benchmarks still depend on succinct, human-annotated text descriptions for each video. These descriptions often fall short in capturing the nuances of video content dynamics and lack stylistic variety in language, constrained by their uniform level of detail and a fixed annotation frequency. As a result, algorithms tend to default to a "memorize the answer" strategy, diverging from the core objective of achieving a deeper understanding of video content. Fortunately, the emergence of large language models (LLMs) has enabled the generation of diverse text. This work utilizes LLMs to generate varied semantic annotations (in terms of text lengths and granularities) for representative SOT benchmarks, thereby establishing a novel multi-modal benchmark. Specifically, we (1) propose a new visual language tracking benchmark with diverse texts, named DTVLT, based on five prominent VLT and SOT benchmarks, including three sub-tasks: short-term tracking, long-term tracking, and global instance tracking. (2) We offer four granularity texts in our benchmark, considering the extent and density of semantic information. We expect this multi-granular generation strategy to foster a favorable environment for VLT and video understanding research. (3) We conduct comprehensive experimental analyses on DTVLT, evaluating the impact of diverse text on tracking performance and hope the identified performance bottlenecks of existing algorithms can support further research in VLT and video understanding. The proposed benchmark, experimental results and toolkit will be released gradually on http://videocube.aitestunion.com/.