# Improving Visual Object Tracking through Visual Prompting arXiv: https://arxiv.org/abs/2409.18901 ยท HTML: https://mlsanity.com/paper/2409.18901/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 65 of its 92 references that are in the map. - First posted on arXiv 27 Sep 2024. ## Builds on (most cited first, 30 of 65) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019) - [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019) - [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017) - [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017) - [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023) - [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018) - [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018) - [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018) - [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021) - [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018) - [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021) - [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019) - [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018) - [Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework](https://mlsanity.com/paper/2203.11991/) (2022) - [A Short Note on the Kinetics-700 Human Action Dataset](https://mlsanity.com/paper/1907.06987/) (2019) - [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018) - [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022) - [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018) - [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020) - [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021) - [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020) - [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023) - [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020) - [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018) ## Built on by (most cited first, 9 of 9) - [Visual Position Prompt for MLLM based Visual Grounding](https://mlsanity.com/paper/2503.15426/) (2025) - [Mamba-FETrack V2: Revisiting State Space Model for Frame-Event based Visual Object Tracking](https://mlsanity.com/paper/2506.23783/) (2025) - [GOT-Edit: Geometry-Aware Generic Object Tracking via Online Model Editing](https://mlsanity.com/paper/2602.08550/) (2026) - [End-to-End Unmixing with Material Prompts for Hyperspectral Object Tracking](https://mlsanity.com/paper/2605.20569/) (2026) - [USS: Unifying Spatial-Semantic Prompting for End to End Embodied Visual Tracking](https://mlsanity.com/paper/2606.25880/) (2026) - [UBATrack: Spatio-Temporal State Space Model for General Multi-Modal Tracking](https://mlsanity.com/paper/2601.14799/) (2026) - [GOT-JEPA: Generic Object Tracking with Model Adaptation and Occlusion Handling using Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2602.14771/) (2026) - [Architecture and evaluation protocol for transformer-based visual object tracking in UAV applications](https://mlsanity.com/paper/2603.03904/) (2026) - [Vision-Language Guided Hyperspectral Object Tracking via Semantics Fusion and Contextual Template Updating](https://mlsanity.com/paper/2606.09167/) (2026) ## Abstract (arXiv) Learning a discriminative model that distinguishes the specified target from surrounding distractors across frames is essential for generic object tracking (GOT). Dynamic adaptation of target representation against distractors remains challenging because prevailing trackers exhibit limited discriminative capability. To address this issue, we present a new visual prompting mechanism for generic object tracking, termed PiVOT. PiVOT introduces mechanisms that leverage the pretrained foundation model (CLIP) to automatically generate and refine visual prompts online, thereby enabling the tracker to suppress distractors through contrastive guidance. To transfer contrastive knowledge from the foundation model to the tracker, PiVOT automatically propagates this knowledge online and dynamically generates and updates visual prompts. Specifically, it proposes a prompt initialization mechanism that produces an initial visual prompt highlighting potential target locations. The foundation model is then used to refine the prompt based on appearance similarities between candidate objects and reference templates across potential targets. After refinement, the visual prompt better highlights potential target locations and reduces irrelevant prompt information. With the proposed prompting mechanism, the tracker can generate instance-aware feature maps guided by the visual prompts, which are incrementally and automatically updated during tracking, thereby effectively suppressing distractors. Extensive experiments across multiple benchmarks indicate that PiVOT, with the proposed prompting mechanism, can suppress distracting objects and improve tracking performance.