# Jointly Modeling Motion and Appearance Cues for Robust RGB-T Tracking

arXiv: https://arxiv.org/abs/2007.02041 · HTML: https://mlsanity.com/paper/2007.02041/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jul 2020.

## Built on by (most cited first, 30 of 48)

- [Visual Prompt Multi-Modal Tracking](https://mlsanity.com/paper/2303.10826/) (2023)
- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2204.04120/) (2022)
- [Bi-directional Adapter for Multi-modal Tracking](https://mlsanity.com/paper/2312.10611/) (2023)
- [SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2403.16002/) (2024)
- [Generative-based Fusion Mechanism for Multi-Modal Tracking](https://mlsanity.com/paper/2309.01728/) (2023)
- [Temporal Adaptive RGBT Tracking with Modality Prompt](https://mlsanity.com/paper/2401.01244/) (2024)
- [Exploring Fusion Strategies for Accurate RGBT Visual Object Tracking](https://mlsanity.com/paper/2201.08673/) (2022)
- [Glass Segmentation with RGB-Thermal Image Pairs](https://mlsanity.com/paper/2204.05453/) (2022)
- [Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2304.14394/) (2023)
- [XTrack: Multimodal Training Boosts RGB-X Video Object Trackers](https://mlsanity.com/paper/2405.17773/) (2024)
- [What You Have is What You Track: Adaptive and Robust Multimodal Tracking](https://mlsanity.com/paper/2507.05899/) (2025)
- [MambaVT: Spatio-Temporal Contextual Modeling for robust RGB-T Tracking](https://mlsanity.com/paper/2408.07889/) (2024)
- [Siamese Infrared and Visible Light Fusion Network for RGB-T Tracking](https://mlsanity.com/paper/2103.07302/) (2021)
- [Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokens](https://mlsanity.com/paper/2401.01674/) (2024)
- [Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation](https://mlsanity.com/paper/2410.11586/) (2024)
- [CADTrack: Learning Contextual Aggregation with Deformable Alignment for Robust RGBT Tracking](https://mlsanity.com/paper/2511.17967/) (2025)
- [CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features](https://mlsanity.com/paper/2505.19434/) (2025)
- [Dynamic Fusion Network for RGBT Tracking](https://mlsanity.com/paper/2109.07662/) (2021)
- [What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging](https://mlsanity.com/paper/2304.13651/) (2023)
- [Global-Local MAV Detection under Challenging Conditions based on Appearance and Motion](https://mlsanity.com/paper/2312.11008/) (2023)
- [RGB-T Tracking Based on Mixed Attention](https://mlsanity.com/paper/2304.04264/) (2023)
- [AFter: Attention-based Fusion Router for RGBT Tracking](https://mlsanity.com/paper/2405.02717/) (2024)
- [X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation](https://mlsanity.com/paper/2409.19342/) (2024)
- [Temporal Aggregation for Adaptive RGBT Tracking](https://mlsanity.com/paper/2201.08949/) (2022)
- [A Survey for Deep RGBT Tracking](https://mlsanity.com/paper/2201.09296/) (2022)
- [Unified Single-Stage Transformer Network for Efficient RGB-T Tracking](https://mlsanity.com/paper/2308.13764/) (2023)
- [RGBT Tracking via Progressive Fusion Transformer with Dynamically Guided Learning](https://mlsanity.com/paper/2303.14778/) (2023)
- [HDRT: A Large-Scale Dataset for Infrared-Guided HDR Imaging](https://mlsanity.com/paper/2406.05475/) (2024)
- [SSF-Net: Spatial-Spectral Fusion Network with Spectral Angle Awareness for Hyperspectral Object Tracking](https://mlsanity.com/paper/2403.05852/) (2024)

## Abstract (arXiv)

In this study, we propose a novel RGB-T tracking framework by jointly modeling both appearance and motion cues. First, to obtain a robust appearance model, we develop a novel late fusion method to infer the fusion weight maps of both RGB and thermal (T) modalities. The fusion weights are determined by using offline-trained global and local multimodal fusion networks, and then adopted to linearly combine the response maps of RGB and T modalities. Second, when the appearance cue is unreliable, we comprehensively take motion cues, i.e., target and camera motions, into account to make the tracker robust. We further propose a tracker switcher to switch the appearance and motion trackers flexibly. Numerous results on three recent RGB-T tracking datasets show that the proposed tracker performs significantly better than other state-of-the-art algorithms.