# Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarks

arXiv: https://arxiv.org/abs/2312.16244 · HTML: https://mlsanity.com/paper/2312.16244/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 25 Dec 2023.

## Builds on (most cited first, 30 of 31)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Visual Prompt Multi-Modal Tracking](https://mlsanity.com/paper/2303.10826/) (2023)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [Convolutional Bypasses Are Better Vision Transformer Adapters](https://mlsanity.com/paper/2207.07039/) (2022)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2204.04120/) (2022)
- [Partial Transfer Learning with Selective Adversarial Networks](https://mlsanity.com/paper/1707.07901/) (2017)
- [Deep Partial Multi-View Learning](https://mlsanity.com/paper/2011.06170/) (2020)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)
- [Multi-Modal Fusion for End-to-End RGB-T Tracking](https://mlsanity.com/paper/1908.11714/) (2019)
- [Prompting for Multi-Modal Tracking](https://mlsanity.com/paper/2207.14571/) (2022)
- [Challenge-Aware RGBT Tracking](https://mlsanity.com/paper/2007.13143/) (2020)
- [Semantic Compositional Networks for Visual Captioning](https://mlsanity.com/paper/1611.08002/) (2016)
- [Bi-directional Adapter for Multi-modal Tracking](https://mlsanity.com/paper/2312.10611/) (2023)
- [ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2106.14591/) (2021)
- [Multi-Adapter RGBT Tracking](https://mlsanity.com/paper/1907.07485/) (2019)
- [Duality-Gated Mutual Condition Network for RGBT Tracking](https://mlsanity.com/paper/2011.07188/) (2020)
- [RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss](https://mlsanity.com/paper/2011.07189/) (2020)

## Built on by (most cited first, 17 of 17)

- [What You Have is What You Track: Adaptive and Robust Multimodal Tracking](https://mlsanity.com/paper/2507.05899/) (2025)
- [M4-SAR: A Multi-Resolution, Multi-Polarization, Multi-Scene, Multi-Source Dataset and Benchmark for optical-SAR Object Detection](https://mlsanity.com/paper/2505.10931/) (2025)
- [CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features](https://mlsanity.com/paper/2505.19434/) (2025)
- [ICPL-ReID: Identity-Conditional Prompt Learning for Multi-Spectral Object Re-Identification](https://mlsanity.com/paper/2505.17821/) (2025)
- [CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework](https://mlsanity.com/paper/2504.12576/) (2025)
- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)
- [RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.03617/) (2026)
- [Human Activity Recognition using RGB-Event based Sensors: A Multi-modal Heat Conduction Model and A Benchmark Dataset](https://mlsanity.com/paper/2504.05830/) (2025)
- [Learning Frequency and Memory-Aware Prompts for Multi-Modal Object Tracking](https://mlsanity.com/paper/2506.23972/) (2025)
- [Spatio-Temporal Conditional Denoising Transformer for Modality-Missing RGBT Tracking](https://mlsanity.com/paper/2607.24701/) (2026)
- [Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking](https://mlsanity.com/paper/2603.13719/) (2026)
- [AnyMod-LLVE: Low-Light Video Enhancement with Modality-Agnostic Inference](https://mlsanity.com/paper/2606.11186/) (2026)
- [Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities](https://mlsanity.com/paper/2606.30597/) (2026)
- [Cross-Modal UAV Object Tracking: State-Aware Representation Learning and A Unified Benchmark](https://mlsanity.com/paper/2607.18768/) (2026)
- [DRGBT-1K: A Large-scale High-quality Benchmark for Dynamic RGBT Tracking](https://mlsanity.com/paper/2607.19772/) (2026)
- [Test-Time Logit Prompting for Source-Free Missing Modality Adaptation](https://mlsanity.com/paper/2609.02039/) (2026)
- [End-to-End Self-Supervised RGB-T Tracking without Modality Misleading](https://mlsanity.com/paper/2609.37162/) (2026)

## Abstract (arXiv)

Current RGBT tracking research relies on the complete multi-modal input, but modal information might miss due to some factors such as thermal sensor self-calibration and data transmission error, called modality-missing challenge in this work. To address this challenge, we propose a novel invertible prompt learning approach, which integrates the content-preserving prompts into a well-trained tracking model to adapt to various modality-missing scenarios, for robust RGBT tracking. Given one modality-missing scenario, we propose to utilize the available modality to generate the prompt of the missing modality to adapt to RGBT tracking model. However, the cross-modality gap between available and missing modalities usually causes semantic distortion and information loss in prompt generation. To handle this issue, we design the invertible prompter by incorporating the full reconstruction of the input available modality from the generated prompt. To provide a comprehensive evaluation platform, we construct several high-quality benchmark datasets, in which various modality-missing scenarios are considered to simulate real-world challenges. Extensive experiments on three modality-missing benchmark datasets show that our method achieves significant performance improvements compared with state-of-the-art methods. We have released the code and simulation datasets at: hrefhttps://github.com/Alexadlu/Modality-missing-RGBT-Tracking.githttps://github.com/Alexadlu/Modality-missing-RGBT-Tracking.git.