mlsanityData to 7 Oct 2026

Visual Prompt Multi-Modal Tracking

arXiv 2303.10826 ↗First posted 20 Mar 2023cs.CVcited by 141

Not on the map. Why?

Abstract (arXiv)

Visible-modal object tracking gives rise to a series of downstream multi-modal tracking tributaries. To inherit the powerful representations of the foundation model, a natural modus operandi for multi-modal tracking is full fine-tuning on the RGB-based parameters. Albeit effective, this manner is not optimal due to the scarcity of downstream data and poor transferability, etc. In this paper, inspired by the recent success of the prompt learning in language models, we develop Visual Prompt multi-modal Tracking (ViPT), which learns the modal-relevant prompts to adapt the frozen pre-trained foundation model to various downstream multimodal tracking tasks. ViPT finds a better way to stimulate the knowledge of the RGB-based model that is pre-trained at scale, meanwhile only introducing a few trainable parameters (less than 1% of model parameters). ViPT outperforms the full fine-tuning paradigm on multiple downstream tracking tasks including RGB+Depth, RGB+Thermal, and RGB+Event tracking. Extensive experiments show the potential of visual prompt learning for multi-modal tracking, and ViPT can achieve state-of-the-art performance while satisfying parameter efficiency. Code and models are available at https://github.com/jiawen-zhu/ViPT.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

40 of 64 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2022Visual Prompt Tuningcited by 1,674
  10. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  11. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
  12. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
28 more
  1. 2018SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networkscited by 445
  2. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 424
  3. 2021Transformer Trackingcited by 412
  4. 2021Learning Spatio-Temporal Transformer for Visual Trackingcited by 348
  5. 2019Learning Discriminative Model Prediction for Trackingcited by 344
  6. 2018TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wildcited by 342
  7. 2022Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Frameworkcited by 328
  8. 2018ATOM: Accurate Tracking by Overlap Maximizationcited by 285
  9. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  10. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 228
  11. 2021VisEvent: Reliable Object Tracking via Collaboration of Frame and Event Flowscited by 128
  12. 2018RGB-T Object Tracking:Benchmark and Baselinecited by 121
  13. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 120
  14. 2021Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Mergingcited by 104
  15. 2021LasHeR: A Large-scale High-diversity Benchmark for RGBT Trackingcited by 94
  16. 2022Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baselinecited by 90
  17. 2017Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensorcited by 84
  18. 2021Object Tracking by Jointly Exploiting Frame and Event Domaincited by 83
  19. 2021DepthTrack : Unveiling the Power of RGBD Trackingcited by 66
  20. 2019Multi-Modal Fusion for End-to-End RGB-T Trackingcited by 63
  21. 2022Prompting for Multi-Modal Trackingcited by 58
  22. 2020Challenge-Aware RGBT Trackingcited by 53
  23. 2022Prompt Vision Transformer for Domain Generalizationcited by 52
  24. 2020Jointly Modeling Motion and Appearance Cues for Robust RGB-T Trackingcited by 47
  25. 2022RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Trackingcited by 42
  26. 2019Dense Feature Aggregation and Pruning for RGBT Trackingcited by 40
  27. 2019DAL -- A Deep Depth-aware Long-term Trackercited by 25
  28. 2022Prompt-Matched Semantic Segmentationcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 45; 2025: 40; 2026: 30.

141 papers.

Sort:

By area:

  1. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  2. 2023Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baselinecited by 73
  3. 2023HIPTrack: Visual Tracking with Historical Promptscited by 71
  4. 2023Single-Model and Any-Modality for Video Object Trackingcited by 64
  5. 2024OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuningcited by 64
  6. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  7. 2024SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Trackingcited by 43
  8. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  9. 2023CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Updatecited by 42
  10. 2023From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videoscited by 39
  11. 2023SAMAug: Point Prompt Augmentation for Segment Anything Modelcited by 38
  12. 2024FaceXFormer: A Unified Transformer for Facial Analysiscited by 34
38 more
  1. 2023Generative-based Fusion Mechanism for Multi-Modal Trackingcited by 33
  2. 2025Towards Universal Modal Tracking with Online Dense Temporal Token Learningcited by 30
  3. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalitiescited by 29
  4. 2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
  5. 2023FreeKD: Knowledge Distillation via Semantic Frequency Promptcited by 27
  6. 2023UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memorycited by 26
  7. 2023Efficient Multimodal Semantic Segmentation via Dual-Prompt Learningcited by 25
  8. 2023Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Cameracited by 21
  9. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  10. 2024Centering the Value of Every Modality: Towards Efficient and Resilient Modality-agnostic Semantic Segmentationcited by 21
  11. 2023Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Trackingcited by 20
  12. 2024XTrack: Multimodal Training Boosts RGB-X Video Object Trackerscited by 20
  13. 2023Instance Tracking in 3D Scenes from Egocentric Videoscited by 17
  14. 2023Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarkscited by 17
  15. 2024CRSOT: Cross-Resolution Object Tracking using Unaligned Frame and Event Camerascited by 17
  16. 2024RGBT Tracking via All-layer Multimodal Interactions with Progressive Fusion Mambacited by 17
  17. 2023DCPT: Darkness Clue-Prompted Tracking in Nighttime UAVscited by 16
  18. 2024WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmarkcited by 15
  19. 2024Towards Zero-shot Point Cloud Anomaly Detection: A Multi-View Projection Frameworkcited by 15
  20. 2025What You Have is What You Track: Adaptive and Robust Multimodal Trackingcited by 15
  21. 2023EZ-CLIP: Efficient Zeroshot Video Action Recognitioncited by 14
  22. 2024Improving Large Models with Small models: Lower Costs and Better Performancecited by 14
  23. 2024Cross-modulated Attention Transformer for RGBT Trackingcited by 14
  24. 2024MambaVT: Spatio-Temporal Contextual Modeling for robust RGB-T Trackingcited by 14
  25. 2024Visible-Thermal Multiple Object Tracking: Large-scale Video Dataset and Progressive Fusion Approachcited by 13
  26. 2023DeNoising-MOT: Towards Multiple Object Tracking with Severe Occlusionscited by 12
  27. 2024MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selectioncited by 12
  28. 2024Transformer RGBT Tracking with Spatio-Temporal Multimodal Tokenscited by 11
  29. 2024Fourier Prompt Tuning for Modality-Incomplete Scene Segmentationcited by 11
  30. 2024Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detectioncited by 11
  31. 2024Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillationcited by 11
  32. 2025Adaptive Perception for Unified Visual Multi-modal Object Trackingcited by 11
  33. 2024Controllable Prompt Tuning For Balancing Group Distributional Robustnesscited by 10
  34. 2024eMoE-Tracker: Environmental MoE-based Transformer for Robust Event-guided Object Trackingcited by 10
  35. 2025CSTrack: Enhancing RGB-X Tracking via Compact Spatiotemporal Featurescited by 10
  36. 2025Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptationcited by 10
  37. 2023Visual Prompt Flexible-Modal Face Anti-Spoofingcited by 9
  38. 2023Unified-modal Salient Object Detection via Adaptive Prompt Learningcited by 9

50 most cited of 141 All 141 (JSON)