# DTLLM-VLT: Diverse Text Generation for Visual Language Tracking Based on LLM

arXiv: https://arxiv.org/abs/2405.12139 · HTML: https://mlsanity.com/paper/2405.12139/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 44 references that are in the map.
- First posted on arXiv 20 May 2024.

## Builds on (most cited first, 25 of 25)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020)
- [Siam R-CNN: Visual Tracking by Re-Detection](https://mlsanity.com/paper/1911.12836/) (2019)
- [Osprey: Pixel Understanding with Visual Instruction Tuning](https://mlsanity.com/paper/2312.10032/) (2023)
- [SiamCAR: Siamese Fully Convolutional Classification and Regression for Visual Tracking](https://mlsanity.com/paper/1911.07241/) (2019)
- [LaSOT: A High-quality Large-scale Single Object Tracking Benchmark](https://mlsanity.com/paper/2009.03465/) (2020)
- [Learning Target Candidate Association to Keep Track of What Not to Track](https://mlsanity.com/paper/2103.16556/) (2021)
- [Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark](https://mlsanity.com/paper/2103.16746/) (2021)
- [A Novel Performance Evaluation Methodology for Single-Target Trackers](https://mlsanity.com/paper/1503.01313/) (2015)
- [Towards Unified Token Learning for Vision-Language Tracking](https://mlsanity.com/paper/2308.14103/) (2023)
- [Joint Visual Grounding and Tracking with Natural Language Specification](https://mlsanity.com/paper/2303.12027/) (2023)
- [Long-term Tracking in the Wild: A Benchmark](https://mlsanity.com/paper/1803.09502/) (2018)
- [Divert More Attention to Vision-Language Tracking](https://mlsanity.com/paper/2207.01076/) (2022)
- [Grounding-Tracking-Integration](https://mlsanity.com/paper/1912.06316/) (2019)
- [Real-time Visual Object Tracking with Natural Language Description](https://mlsanity.com/paper/1907.11751/) (2019)
- [Describe and Attend to Track: Learning Natural Language guided Structural Representation and Visual Attention for Object Tracking](https://mlsanity.com/paper/1811.10014/) (2018)
- [Global Instance Tracking: Locating Target More Like Humans](https://mlsanity.com/paper/2202.13073/) (2022)
- [Siamese Natural Language Tracker: Tracking by Natural Language Descriptions with Siamese Trackers](https://mlsanity.com/paper/1912.02048/) (2019)
- [BioDrone: A Bionic Drone-based Single Object Tracking Benchmark for Robust Vision](https://mlsanity.com/paper/2402.04519/) (2024)
- [SOTVerse: A User-defined Task Space of Single Object Tracking](https://mlsanity.com/paper/2204.07414/) (2022)

## Built on by (most cited first, 24 of 24)

- [UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility](https://mlsanity.com/paper/2501.02341/) (2025)
- [NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generation](https://mlsanity.com/paper/2507.11245/) (2025)
- [WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark](https://mlsanity.com/paper/2405.19818/) (2024)
- [DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM](https://mlsanity.com/paper/2410.02492/) (2024)
- [Visual Language Tracking with Multi-modal Interaction: A Robust Benchmark](https://mlsanity.com/paper/2409.08887/) (2024)
- [How Texts Help? A Fine-grained Evaluation to Reveal the Role of Language in Vision-Language Tracking](https://mlsanity.com/paper/2411.15600/) (2024)
- [CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features](https://mlsanity.com/paper/2505.19434/) (2025)
- [ATCTrack: Aligning Target-Context Cues with Dynamic Target States for Robust Vision-Language Tracking](https://mlsanity.com/paper/2507.19875/) (2025)
- [Enhancing Vision-Language Tracking by Effectively Converting Textual Cues into Visual Cues](https://mlsanity.com/paper/2412.19648/) (2024)
- [Enhancing LLM-based Autonomous Driving Agents to Mitigate Perception Attacks](https://mlsanity.com/paper/2409.14488/) (2024)
- [Do Large Language Models Possess Sensitive to Sentiment?](https://mlsanity.com/paper/2409.02370/) (2024)
- [Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness](https://mlsanity.com/paper/2504.16936/) (2025)
- [DARTer: Dynamic Adaptive Representation Tracker for Nighttime UAV Tracking](https://mlsanity.com/paper/2505.00752/) (2025)
- [DiffoRA: Enabling Parameter-Efficient Fine-Tuning via Differential Module Selection](https://mlsanity.com/paper/2502.08905/) (2025)
- [ATSTrack: Enhancing Visual-Language Tracking by Aligning Temporal and Spatial Scales](https://mlsanity.com/paper/2507.00454/) (2025)
- [AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data](https://mlsanity.com/paper/2507.18442/) (2025)
- [RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.03617/) (2026)
- [4DVLT: Dynamic Scene Understanding with Worldline-Centered Vision-Language Tracking](https://mlsanity.com/paper/2606.22631/) (2026)
- [BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching](https://mlsanity.com/paper/2410.18701/) (2024)
- [Learning to Track Instance from Single Nature Language Description](https://mlsanity.com/paper/2605.07064/) (2026)
- [Revisiting LLM Adaptation for 3D CT Report Generation: A Study of Scaling and Diagnostic Priors](https://mlsanity.com/paper/2606.17213/) (2026)
- [Maintaining Informative Coherence: Migrating Hallucinations in Large Language Models via Absorbing Markov Chains](https://mlsanity.com/paper/2410.20340/) (2024)
- [SOI is the Root of All Evil: Quantifying and Breaking Similar Object Interference in Single Object Tracking](https://mlsanity.com/paper/2508.09524/) (2025)
- [TalkMatrix: Generating Character Dialogue that is Both Consistent and Diverse](https://mlsanity.com/paper/2609.19022/) (2026)

## Abstract (arXiv)

Visual Language Tracking (VLT) enhances single object tracking (SOT) by integrating natural language descriptions from a video, for the precise tracking of a specified object. By leveraging high-level semantic information, VLT guides object tracking, alleviating the constraints associated with relying on a visual modality. Nevertheless, most VLT benchmarks are annotated in a single granularity and lack a coherent semantic framework to provide scientific guidance. Moreover, coordinating human annotators for high-quality annotations is laborious and time-consuming. To address these challenges, we introduce DTLLM-VLT, which automatically generates extensive and multi-granularity text to enhance environmental diversity. (1) DTLLM-VLT generates scientific and multi-granularity text descriptions using a cohesive prompt framework. Its succinct and highly adaptable design allows seamless integration into various visual tracking benchmarks. (2) We select three prominent benchmarks to deploy our approach: short-term tracking, long-term tracking, and global instance tracking. We offer four granularity combinations for these benchmarks, considering the extent and density of semantic information, thereby showcasing the practicality and versatility of DTLLM-VLT. (3) We conduct comparative experiments on VLT benchmarks with different text granularities, evaluating and analyzing the impact of diverse text on tracking performance. Conclusionally, this work leverages LLM to provide multi-granularity semantic information for VLT task from efficient and diverse perspectives, enabling fine-grained evaluation of multi-modal trackers. In the future, we believe this work can be extended to more datasets to support vision datasets understanding.