# Prompting for Multi-Modal Tracking

arXiv: https://arxiv.org/abs/2207.14571 · HTML: https://mlsanity.com/paper/2207.14571/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 29 Jul 2022.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [Learning Target Candidate Association to Keep Track of What Not to Track](https://mlsanity.com/paper/2103.16556/) (2021)
- [VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flows](https://mlsanity.com/paper/2108.05015/) (2021)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking](https://mlsanity.com/paper/2104.13202/) (2021)
- [Alpha-Refine: Boosting Tracking Performance by Precise Bounding Box Estimation](https://mlsanity.com/paper/2007.02024/) (2020)
- [DepthTrack : Unveiling the Power of RGBD Tracking](https://mlsanity.com/paper/2108.13962/) (2021)
- [Multi-Modal Fusion for End-to-End RGB-T Tracking](https://mlsanity.com/paper/1908.11714/) (2019)
- [Object Tracking by Reconstruction with View-Specific Discriminative Correlation Filters](https://mlsanity.com/paper/1811.10863/) (2018)
- [CDTB: A Color and Depth Visual Object Tracking Dataset and Benchmark](https://mlsanity.com/paper/1907.00618/) (2019)
- [DAL -- A Deep Depth-aware Long-term Tracker](https://mlsanity.com/paper/1912.00660/) (2019)
- [Asynchronous Tracking-by-Detection on Adaptive Time Surfaces for Event-based Object Tracking](https://mlsanity.com/paper/2002.05583/) (2020)
- [TSDM: Tracking by SiamRPN++ with a Depth-refiner and a Mask-generator](https://mlsanity.com/paper/2005.04063/) (2020)

## Built on by (most cited first, 30 of 59)

- [Visual Prompt Multi-Modal Tracking](https://mlsanity.com/paper/2303.10826/) (2023)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Single-Model and Any-Modality for Video Object Tracking](https://mlsanity.com/paper/2311.15851/) (2023)
- [OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning](https://mlsanity.com/paper/2403.09634/) (2024)
- [Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model](https://mlsanity.com/paper/2208.08340/) (2022)
- [SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2403.16002/) (2024)
- [Generative-based Fusion Mechanism for Multi-Modal Tracking](https://mlsanity.com/paper/2309.01728/) (2023)
- [Towards Universal Modal Tracking with Online Dense Temporal Token Learning](https://mlsanity.com/paper/2507.20177/) (2025)
- [Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning](https://mlsanity.com/paper/2312.00360/) (2023)
- [Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking](https://mlsanity.com/paper/2304.14394/) (2023)
- [XTrack: Multimodal Training Boosts RGB-X Video Object Trackers](https://mlsanity.com/paper/2405.17773/) (2024)
- [Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarks](https://mlsanity.com/paper/2312.16244/) (2023)
- [CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Cameras](https://mlsanity.com/paper/2401.02826/) (2024)
- [Autogenic Language Embedding for Coherent Point Tracking](https://mlsanity.com/paper/2407.20730/) (2024)
- [RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mamba](https://mlsanity.com/paper/2408.08827/) (2024)
- [What You Have is What You Track: Adaptive and Robust Multimodal Tracking](https://mlsanity.com/paper/2507.05899/) (2025)
- [MambaVT: Spatio-Temporal Contextual Modeling for robust RGB-T Tracking](https://mlsanity.com/paper/2408.07889/) (2024)
- [Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation](https://mlsanity.com/paper/2410.11586/) (2024)
- [Adaptive Perception for Unified Visual Multi-modal Object Tracking](https://mlsanity.com/paper/2502.06583/) (2025)
- [POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View World](https://mlsanity.com/paper/2403.05856/) (2024)
- [CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Features](https://mlsanity.com/paper/2505.19434/) (2025)
- [Visual Prompt Flexible-Modal Face Anti-Spoofing](https://mlsanity.com/paper/2307.13958/) (2023)
- [TENet: Targetness Entanglement Incorporating with Multi-Scale Pooling and Mutually-Guided Fusion for RGB-E Object Tracking](https://mlsanity.com/paper/2405.05004/) (2024)
- [Reliable Object Tracking by Multimodal Hybrid Feature Extraction and Transformer-Based Fusion](https://mlsanity.com/paper/2405.17903/) (2024)
- [AFter: Attention-based Fusion Router for RGBT Tracking](https://mlsanity.com/paper/2405.02717/) (2024)
- [X-Prompt: Multi-modal Visual Prompt for Video Object Segmentation](https://mlsanity.com/paper/2409.19342/) (2024)
- [SSFam: Scribble Supervised Salient Object Detection Family](https://mlsanity.com/paper/2409.04817/) (2024)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [Transformer-based RGB-T Tracking with Channel and Spatial Feature Fusion](https://mlsanity.com/paper/2405.03177/) (2024)
- [Efficient Motion Prompt Learning for Robust Visual Tracking](https://mlsanity.com/paper/2505.16321/) (2025)

## Abstract (arXiv)

Multi-modal tracking gains attention due to its ability to be more accurate and robust in complex scenarios compared to traditional RGB-based tracking. Its key lies in how to fuse multi-modal data and reduce the gap between modalities. However, multi-modal tracking still severely suffers from data deficiency, thus resulting in the insufficient learning of fusion modules. Instead of building such a fusion module, in this paper, we provide a new perspective on multi-modal tracking by attaching importance to the multi-modal visual prompts. We design a novel multi-modal prompt tracker (ProTrack), which can transfer the multi-modal inputs to a single modality by the prompt paradigm. By best employing the tracking ability of pre-trained RGB trackers learning at scale, our ProTrack can achieve high-performance multi-modal tracking by only altering the inputs, even without any extra training on multi-modal data. Extensive experiments on 5 benchmark datasets demonstrate the effectiveness of the proposed ProTrack.