# Context-Aware Integration of Language and Visual References for Natural Language Tracking

arXiv: https://arxiv.org/abs/2403.19975 · HTML: https://mlsanity.com/paper/2403.19975/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2024.

## Built on by (most cited first, 22 of 22)

- [MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation](https://mlsanity.com/paper/2512.10945/) (2025)
- [RefMask3D: Language-Guided Transformer for 3D Referring Segmentation](https://mlsanity.com/paper/2407.18244/) (2024)
- [DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM](https://mlsanity.com/paper/2410.02492/) (2024)
- [How Texts Help? A Fine-grained Evaluation to Reveal the Role of Language in Vision-Language Tracking](https://mlsanity.com/paper/2411.15600/) (2024)
- [ATCTrack: Aligning Target-Context Cues with Dynamic Target States for Robust Vision-Language Tracking](https://mlsanity.com/paper/2507.19875/) (2025)
- [ReasoningTrack: Chain-of-Thought Reasoning for Long-term Vision-Language Tracking](https://mlsanity.com/paper/2508.05221/) (2025)
- [Enhancing Vision-Language Tracking by Effectively Converting Textual Cues into Visual Cues](https://mlsanity.com/paper/2412.19648/) (2024)
- [VPTracker: Global Vision-Language Tracking via Visual Prompt](https://mlsanity.com/paper/2512.22799/) (2025)
- [CLDTracker: A Comprehensive Language Description for Visual Tracking](https://mlsanity.com/paper/2505.23704/) (2025)
- [ATSTrack: Enhancing Visual-Language Tracking by Aligning Temporal and Spatial Scales](https://mlsanity.com/paper/2507.00454/) (2025)
- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)
- [Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2510.19802/) (2025)
- [UniSOT: A Unified Framework for Multi-Modality Single Object Tracking](https://mlsanity.com/paper/2511.01427/) (2025)
- [4DVLT: Dynamic Scene Understanding with Worldline-Centered Vision-Language Tracking](https://mlsanity.com/paper/2606.22631/) (2026)
- [Towards General Multimodal Visual Tracking](https://mlsanity.com/paper/2503.11218/) (2025)
- [EPIPTrack: Rethinking Prompt Modeling with Explicit and Implicit Prompts for Multi-Object Tracking](https://mlsanity.com/paper/2510.13235/) (2025)
- [GLAD: Generative Language-Assisted Visual Tracking for Low-Semantic Templates](https://mlsanity.com/paper/2602.00570/) (2026)
- [Learning to Track Instance from Single Nature Language Description](https://mlsanity.com/paper/2605.07064/) (2026)
- [SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking](https://mlsanity.com/paper/2606.24449/) (2026)
- [Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation](https://mlsanity.com/paper/2604.01974/) (2026)
- [Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking](https://mlsanity.com/paper/2606.29357/) (2026)
- [Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking](https://mlsanity.com/paper/2608.29126/) (2026)

## Abstract (arXiv)

Tracking by natural language specification (TNL) aims to consistently localize a target in a video sequence given a linguistic description in the initial frame. Existing methodologies perform language-based and template-based matching for target reasoning separately and merge the matching results from two sources, which suffer from tracking drift when language and visual templates miss-align with the dynamic target state and ambiguity in the later merging stage. To tackle the issues, we propose a joint multi-modal tracking framework with 1) a prompt modulation module to leverage the complementarity between temporal visual templates and language expressions, enabling precise and context-aware appearance and linguistic cues, and 2) a unified target decoding module to integrate the multi-modal reference cues and executes the integrated queries on the search image to predict the target location in an end-to-end manner directly. This design ensures spatio-temporal consistency by leveraging historical visual information and introduces an integrated solution, generating predictions in a single step. Extensive experiments conducted on TNL2K, OTB-Lang, LaSOT, and RefCOCOg validate the efficacy of our proposed approach. The results demonstrate competitive performance against state-of-the-art methods for both tracking and grounding.