# Few-Shot Backdoor Attacks on Visual Object Tracking

arXiv: https://arxiv.org/abs/2201.13178 · HTML: https://mlsanity.com/paper/2201.13178/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 47 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks](https://mlsanity.com/paper/1805.12185/) (2018)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [STRIP: A Defence Against Trojan Attacks on Deep Neural Networks](https://mlsanity.com/paper/1902.06531/) (2019)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020)
- [Invisible Backdoor Attack with Sample-Specific Triggers](https://mlsanity.com/paper/2012.03816/) (2020)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Anti-Backdoor Learning: Training Clean Models on Poisoned Data](https://mlsanity.com/paper/2110.11571/) (2021)
- [Adversarial Neuron Pruning Purifies Backdoored Deep Models](https://mlsanity.com/paper/2110.14430/) (2021)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines](https://mlsanity.com/paper/1911.06188/) (2019)
- [Clean-Label Backdoor Attacks on Video Recognition Models](https://mlsanity.com/paper/2003.03030/) (2020)
- [Adversarial Unlearning of Backdoors via Implicit Hypergradient](https://mlsanity.com/paper/2110.03735/) (2021)
- [Rethinking the Backdoor Attacks' Triggers: A Frequency Perspective](https://mlsanity.com/paper/2104.03413/) (2021)
- [Neural Trojans](https://mlsanity.com/paper/1710.00942/) (2017)
- [Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness](https://mlsanity.com/paper/2005.00060/) (2020)
- [Robust Anomaly Detection and Backdoor Attack Detection Via Differential Privacy](https://mlsanity.com/paper/1911.07116/) (2019)
- [Deep Feature Space Trojan Attack of Neural Networks by Controlled Detoxification](https://mlsanity.com/paper/2012.11212/) (2020)

## Built on by (most cited first, 30 of 35)

- [Backdoor Defense via Decoupling the Training Process](https://mlsanity.com/paper/2202.03423/) (2022)
- [Untargeted Backdoor Watermark: Towards Harmless and Stealthy Dataset Copyright Protection](https://mlsanity.com/paper/2210.00875/) (2022)
- [Black-box Dataset Ownership Verification via Backdoor Watermarking](https://mlsanity.com/paper/2209.06015/) (2022)
- [Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models](https://mlsanity.com/paper/2210.09545/) (2022)
- [Domain Watermark: Effective and Harmless Dataset Copyright Protection is Closed at Hand](https://mlsanity.com/paper/2310.14942/) (2023)
- [MM-BD: Post-Training Detection of Backdoor Attacks with Arbitrary Backdoor Pattern Types Using a Maximum Margin Statistic](https://mlsanity.com/paper/2205.06900/) (2022)
- [Black-box Backdoor Defense via Zero-shot Image Purification](https://mlsanity.com/paper/2303.12175/) (2023)
- [Backdoor Defense via Deconfounded Representation Learning](https://mlsanity.com/paper/2303.06818/) (2023)
- [Poisoned Forgery Face: Towards Backdoor Attacks on Face Forgery Detection](https://mlsanity.com/paper/2402.11473/) (2024)
- [Untargeted Backdoor Attack against Object Detection](https://mlsanity.com/paper/2211.05638/) (2022)
- [REFINE: Inversion-Free Backdoor Defense via Model Reprogramming](https://mlsanity.com/paper/2502.18508/) (2025)
- [MOVE: Effective and Harmless Ownership Verification via Embedded External Features](https://mlsanity.com/paper/2208.02820/) (2022)
- [Distilling Cognitive Backdoor Patterns within an Image](https://mlsanity.com/paper/2301.10908/) (2023)
- [Backdoor Attack with Sparse and Invisible Trigger](https://mlsanity.com/paper/2306.06209/) (2023)
- [PointNCBW: Towards Dataset Ownership Verification for Point Clouds via Negative Clean-label Backdoor Watermark](https://mlsanity.com/paper/2408.05500/) (2024)
- [PolicyCleanse: Backdoor Detection and Mitigation in Reinforcement Learning](https://mlsanity.com/paper/2202.03609/) (2022)
- [Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypots](https://mlsanity.com/paper/2310.18633/) (2023)
- [UMD: Unsupervised Model Detection for X2X Backdoor Attacks](https://mlsanity.com/paper/2305.18651/) (2023)
- [Nearest is Not Dearest: Towards Practical Defense against Quantization-conditioned Backdoor Attacks](https://mlsanity.com/paper/2405.12725/) (2024)
- [Backdoor Cleansing with Unlabeled Data](https://mlsanity.com/paper/2211.12044/) (2022)
- [On the Effectiveness of Adversarial Training against Backdoor Attacks](https://mlsanity.com/paper/2202.10627/) (2022)
- [Backdoor Attacks on Crowd Counting](https://mlsanity.com/paper/2207.05641/) (2022)
- [Backdoor Defense via Suppressing Model Shortcuts](https://mlsanity.com/paper/2211.05631/) (2022)
- [Backdoor Attacks on Time Series: A Generative Approach](https://mlsanity.com/paper/2211.07915/) (2022)
- [Fake the Real: Backdoor Attack on Deep Speech Classification via Voice Conversion](https://mlsanity.com/paper/2306.15875/) (2023)
- [Temporal Unlearnable Examples: Preventing Personal Video Data from Unauthorized Exploitation by Object Tracking](https://mlsanity.com/paper/2507.07483/) (2025)
- [Diffusion Theory as a Scalpel: Detecting and Purifying Poisonous Dimensions in Pre-trained Language Models Caused by Backdoor or Bias](https://mlsanity.com/paper/2305.04547/) (2023)
- [VSVC: Backdoor attack against Keyword Spotting based on Voiceprint Selection and Voice Conversion](https://mlsanity.com/paper/2212.10103/) (2022)
- [Physical Backdoor: Towards Temperature-based Backdoor Attacks in the Physical World](https://mlsanity.com/paper/2404.19417/) (2024)
- [Invisible Backdoor Attack with Dynamic Triggers against Person Re-identification](https://mlsanity.com/paper/2211.10933/) (2022)

## Abstract (arXiv)

Visual object tracking (VOT) has been widely adopted in mission-critical applications, such as autonomous driving and intelligent surveillance systems. In current practice, third-party resources such as datasets, backbone networks, and training platforms are frequently used to train high-performance VOT models. Whilst these resources bring certain convenience, they also introduce new security threats into VOT models. In this paper, we reveal such a threat where an adversary can easily implant hidden backdoors into VOT models by tempering with the training process. Specifically, we propose a simple yet effective few-shot backdoor attack (FSBA) that optimizes two losses alternately: 1) a feature loss defined in the hidden feature space, and 2) the standard tracking loss. We show that, once the backdoor is embedded into the target model by our FSBA, it can trick the model to lose track of specific objects even when the trigger only appears in one or a few frames. We examine our attack in both digital and physical-world settings and show that it can significantly degrade the performance of state-of-the-art VOT trackers. We also show that our attack is resistant to potential defenses, highlighting the vulnerability of VOT models to potential backdoor attacks.