# Vision-Language Guided Hyperspectral Object Tracking via Semantics Fusion and Contextual Template Updating

arXiv: https://arxiv.org/abs/2606.09167 · HTML: https://mlsanity.com/paper/2606.09167/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 72 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [LocalMamba: Visual State Space Model with Windowed Selective Scan](https://mlsanity.com/paper/2403.09338/) (2024)
- [Autoregressive Queries for Adaptive Tracking with Spatio-TemporalTransformers](https://mlsanity.com/paper/2403.10574/) (2024)
- [Divert More Attention to Vision-Language Tracking](https://mlsanity.com/paper/2207.01076/) (2022)
- [Exploring Enhanced Contextual Information for Video-Level Object Tracking](https://mlsanity.com/paper/2412.11023/) (2024)
- [Material Based Object Tracking in Hyperspectral Videos: Benchmark and Algorithms](https://mlsanity.com/paper/1812.04179/) (2018)
- [CiteTracker: Correlating Image and Text for Visual Tracking](https://mlsanity.com/paper/2308.11322/) (2023)
- [All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment](https://mlsanity.com/paper/2307.03373/) (2023)
- [Language-aware Domain Generalization Network for Cross-Scene Hyperspectral Image Classification](https://mlsanity.com/paper/2209.02700/) (2022)
- [DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM](https://mlsanity.com/paper/2410.02492/) (2024)
- [How Texts Help? A Fine-grained Evaluation to Reveal the Role of Language in Vision-Language Tracking](https://mlsanity.com/paper/2411.15600/) (2024)
- [Machine learning based hyperspectral image analysis: A survey](https://mlsanity.com/paper/1802.08701/) (2018)
- [Improving Visual Object Tracking through Visual Prompting](https://mlsanity.com/paper/2409.18901/) (2024)
- [Enhancing Vision-Language Tracking by Effectively Converting Textual Cues into Visual Cues](https://mlsanity.com/paper/2412.19648/) (2024)
- [Hy-Tracker: A Novel Framework for Enhancing Efficiency and Accuracy of Object Tracking in Hyperspectral Videos](https://mlsanity.com/paper/2311.18199/) (2023)
- [Object Tracking in Hyperspectral Videos with Convolutional Features and Kernelized Correlation Filter](https://mlsanity.com/paper/1810.11819/) (2018)
- [SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness for Hyperspectral Object Tracking](https://mlsanity.com/paper/2403.05852/) (2024)
- [Supervised Embedded Methods for Hyperspectral Band Selection](https://mlsanity.com/paper/2401.11420/) (2024)
- [Hyperspectral Mamba for Hyperspectral Object Tracking](https://mlsanity.com/paper/2509.08265/) (2025)

## Abstract (arXiv)

Hyperspectral object tracking (HOT) leverages the rich spectral information provided by hyperspectral videos (HSVs), offering substantial potential for object tracking. However, efficiently extracting and exploiting spectral information from redundant spectral bands remains a fundamental challenge, which severely limits model generalization and tracking performance. Moreover, in dynamic scenes, targets often experience drastic appearance variations due to factors such as occlusion and illumination changes. These variations lead to large deformations between the current frame and the template. Such discrepancies pose major challenges for existing temporal modeling approaches. In this work, we propose VLHTrack, a novel hyperspectral vision-language (VL) joint tracking framework. Specifically, we incorporate language priors to address the fundamental challenge of spectral redundancy by designing a Language-Guided Band Selection Module (LBSM). By leveraging Large Language Model (LLM) descriptions, LBSM establishes a semantic-to-spectral mapping that mitigates redundancy and accentuates discriminative spectral features. A Multi-Modal Vision-Language Fusion Module is then employed to seamlessly integrate visual and linguistic embeddings, harnessing their complementary advantages to learn coherent cross-modal representations. To address target deformation in long-term sequences, we propose a dynamic update template feature strategy implemented via the Dynamic Template Update with Mamba (DTUM) module. By leveraging selective state space modeling, DTUM learns inter-frame dependencies to update template feature, ensuring efficient template feature evolution guided by temporal context. Experiments on HOT2023 and HOT2024 demonstrate that VLHTrack outperforms state-of-the-art (SOTA) methods.