# Visual Object Tracking across Diverse Data Modalities: A Review

arXiv: https://arxiv.org/abs/2412.09991 · HTML: https://mlsanity.com/paper/2412.09991/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 443 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](https://mlsanity.com/paper/2205.13790/) (2022)
- [ODTrack: Online Dense Temporal Token Learning for Visual Tracking](https://mlsanity.com/paper/2401.01686/) (2024)
- [Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking](https://mlsanity.com/paper/2203.05328/) (2022)
- [HIPTrack: Visual Tracking with Historical Prompts](https://mlsanity.com/paper/2311.02072/) (2023)
- [Explicit Visual Prompts for Visual Object Tracking](https://mlsanity.com/paper/2401.03142/) (2024)
- [3D Object Tracking with Transformer](https://mlsanity.com/paper/2110.14921/) (2021)
- [Divert More Attention to Vision-Language Tracking](https://mlsanity.com/paper/2207.01076/) (2022)
- [Bi-directional Adapter for Multi-modal Tracking](https://mlsanity.com/paper/2312.10611/) (2023)
- [3D Siamese Transformer Network for Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2207.11995/) (2022)
- [SparseTT: Visual Tracking with Sparse Transformers](https://mlsanity.com/paper/2205.03776/) (2022)
- [MFGNet: Dynamic Modality-Aware Filter Generation for RGB-T Tracking](https://mlsanity.com/paper/2107.10433/) (2021)
- [Unifying Visual and Vision-Language Tracking via Contrastive Learning](https://mlsanity.com/paper/2401.11228/) (2024)
- [Learning Target-aware Representation for Visual Tracking via Informative Interactions](https://mlsanity.com/paper/2201.02526/) (2022)
- [Describe and Attend to Track: Learning Natural Language guided Structural Representation and Visual Attention for Object Tracking](https://mlsanity.com/paper/1811.10014/) (2018)
- [Efficient Bird Eye View Proposals for 3D Siamese Tracking](https://mlsanity.com/paper/1903.10168/) (2019)
- [Exploring Point-BEV Fusion for 3D Point Cloud Object Tracking with Transformer](https://mlsanity.com/paper/2208.05216/) (2022)
- [OSP2B: One-Stage Point-to-Box Network for 3D Siamese Tracking](https://mlsanity.com/paper/2304.11584/) (2023)
- [MaIL: A Unified Mask-Image-Language Trimodal Network for Referring Image Segmentation](https://mlsanity.com/paper/2111.10747/) (2021)
- [Towards Category Unification of 3D Single Object Tracking on Point Clouds](https://mlsanity.com/paper/2401.11204/) (2024)
- [Multi-modal Visual Tracking: Review and Experimental Comparison](https://mlsanity.com/paper/2012.04176/) (2020)
- [Siamese Infrared and Visible Light Fusion Network for RGB-T Tracking](https://mlsanity.com/paper/2103.07302/) (2021)
- [RGBD Object Tracking: An In-depth Review](https://mlsanity.com/paper/2203.14134/) (2022)
- [Divert More Attention to Vision-Language Object Tracking](https://mlsanity.com/paper/2307.10046/) (2023)
- [M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognition](https://mlsanity.com/paper/2401.11649/) (2024)
- [M3SOT: Multi-frame, Multi-field, Multi-space 3D Single Object Tracking](https://mlsanity.com/paper/2312.06117/) (2023)
- [Dynamic Fusion Network for RGBT Tracking](https://mlsanity.com/paper/2109.07662/) (2021)

## Built on by (most cited first, 2 of 2)

- [O-MaMa: Learning Object Mask Matching between Egocentric and Exocentric Views](https://mlsanity.com/paper/2506.06026/) (2025)
- [RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.03617/) (2026)

## Abstract (arXiv)

Visual Object Tracking (VOT) is an attractive and significant research area in computer vision, which aims to recognize and track specific targets in video sequences where the target objects are arbitrary and class-agnostic. The VOT technology could be applied in various scenarios, processing data of diverse modalities such as RGB, thermal infrared and point cloud. Besides, since no one sensor could handle all the dynamic and varying environments, multi-modal VOT is also investigated. This paper presents a comprehensive survey of the recent progress of both single-modal and multi-modal VOT, especially the deep learning methods. Specifically, we first review three types of mainstream single-modal VOT, including RGB, thermal infrared and point cloud tracking. In particular, we conclude four widely-used single-modal frameworks, abstracting their schemas and categorizing the existing inheritors. Then we summarize four kinds of multi-modal VOT, including RGB-Depth, RGB-Thermal, RGB-LiDAR and RGB-Language. Moreover, the comparison results in plenty of VOT benchmarks of the discussed modalities are presented. Finally, we provide recommendations and insightful observations, inspiring the future development of this fast-growing literature.