# Awesome Multi-modal Object Tracking

arXiv: https://arxiv.org/abs/2405.14200 · HTML: https://mlsanity.com/paper/2405.14200/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 13 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 8 of 8)

- [Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark](https://mlsanity.com/paper/2103.16746/) (2021)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [DepthTrack : Unveiling the Power of RGBD Tracking](https://mlsanity.com/paper/2108.13962/) (2021)
- [WebUAV-3M: A Benchmark for Unveiling the Power of Million-Scale Deep UAV Tracking](https://mlsanity.com/paper/2201.07425/) (2022)
- [CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Cameras](https://mlsanity.com/paper/2401.02826/) (2024)
- [Multi-modal Visual Tracking: Review and Experimental Comparison](https://mlsanity.com/paper/2012.04176/) (2020)
- [RGBD Object Tracking: An In-depth Review](https://mlsanity.com/paper/2203.14134/) (2022)
- [A Survey for Deep RGBT Tracking](https://mlsanity.com/paper/2201.09296/) (2022)

## Built on by (most cited first, 7 of 7)

- [WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark](https://mlsanity.com/paper/2405.19818/) (2024)
- [Underwater Camouflaged Object Tracking Meets Vision-Language SAM2](https://mlsanity.com/paper/2409.16902/) (2024)
- [COST: Contrastive One-Stage Transformer for Vision-Language Small Object Tracking](https://mlsanity.com/paper/2504.01321/) (2025)
- [Mamba-FETrack V2: Revisiting State Space Model for Frame-Event based Visual Object Tracking](https://mlsanity.com/paper/2506.23783/) (2025)
- [Omni Survey for Multimodality Analysis in Visual Object Tracking](https://mlsanity.com/paper/2508.13000/) (2025)
- [SwiTrack: Tri-State Switch for Cross-Modal Object Tracking](https://mlsanity.com/paper/2511.16227/) (2025)
- [How Far are Modern Trackers from UAV-Anti-UAV? A Million-Scale Benchmark and New Baseline](https://mlsanity.com/paper/2512.07385/) (2025)

## Abstract (arXiv)

Multi-modal object tracking (MMOT) is an emerging field that combines data from various modalities, eg vision (RGB), depth, thermal infrared, event, language and audio, to estimate the state of an arbitrary object in a video sequence. It is of great significance for many applications such as autonomous driving and intelligent surveillance. In recent years, MMOT has received more and more attention. However, existing MMOT algorithms mainly focus on two modalities (eg RGB+depth, RGB+thermal infrared, and RGB+language). To leverage more modalities, some recent efforts have been made to learn a unified visual object tracking model for any modality. Additionally, some large-scale multi-modal tracking benchmarks have been established by simultaneously providing more than two modalities, such as vision-language-audio (eg WebUAV-3M) and vision-depth-language (eg UniMod1K). To track the latest progress in MMOT, we conduct a comprehensive investigation in this report. Specifically, we first divide existing MMOT tasks into five main categories, ie RGBL tracking, RGBE tracking, RGBD tracking, RGBT tracking, and miscellaneous (RGB+X), where X can be any modality, such as language, depth, and event. Then, we analyze and summarize each MMOT task, focusing on widely used datasets and mainstream tracking algorithms based on their technical paradigms (eg self-supervised learning, prompt learning, knowledge distillation, generative models, and state space models). Finally, we maintain a continuously updated paper list for MMOT at https://github.com/983632847/Awesome-Multimodal-Object-Tracking.