# Efficient Decision-based Black-box Patch Attacks on Video Recognition

arXiv: https://arxiv.org/abs/2303.11917 · HTML: https://mlsanity.com/paper/2303.11917/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 61 references that are in the map.
- First posted on arXiv 21 Mar 2023.

## Builds on (most cited first, 30 of 38)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [One pixel attack for fooling deep neural networks](https://mlsanity.com/paper/1710.08864/) (2017)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [Black-box Adversarial Attacks with Limited Queries and Information](https://mlsanity.com/paper/1804.08598/) (2018)
- [Efficient Decision-based Black-box Adversarial Attacks on Face Recognition](https://mlsanity.com/paper/1904.04433/) (2019)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [LaVAN: Localized and Visible Adversarial Noise](https://mlsanity.com/paper/1801.02608/) (2018)
- [Local Gradients Smoothing: Defense against localized adversarial attacks](https://mlsanity.com/paper/1807.01216/) (2018)
- [PatchAttack: A Black-box Texture-based Attack with Reinforcement Learning](https://mlsanity.com/paper/2004.05682/) (2020)
- [Efficient Adversarial Attacks for Visual Object Tracking](https://mlsanity.com/paper/2008.00217/) (2020)
- [Sparse-RS: a versatile framework for query-efficient sparse black-box adversarial attacks](https://mlsanity.com/paper/2006.12834/) (2020)
- [T-SEA: Transfer-based Self-Ensemble Attack on Object Detection](https://mlsanity.com/paper/2211.09773/) (2022)
- [Black-box Adversarial Attacks on Video Recognition Models](https://mlsanity.com/paper/1904.05181/) (2019)
- [Sparse Adversarial Perturbations for Videos](https://mlsanity.com/paper/1803.02536/) (2018)
- [Adv-watermark: A Novel Watermark Perturbation for Adversarial Examples](https://mlsanity.com/paper/2008.01919/) (2020)
- [CMUA-Watermark: A Cross-Model Universal Adversarial Watermark for Combating Deepfakes](https://mlsanity.com/paper/2105.10872/) (2021)
- [Towards Practical Certifiable Patch Defense with Vision Transformer](https://mlsanity.com/paper/2203.08519/) (2022)
- [Heuristic Black-box Adversarial Attacks on Video Recognition Models](https://mlsanity.com/paper/1911.09449/) (2019)
- [Boosting the Transferability of Adversarial Attacks with Global Momentum Initialization](https://mlsanity.com/paper/2211.11236/) (2022)
- [Adversarial Attacks on Black Box Video Classifiers: Leveraging the Power of Geometric Transformations](https://mlsanity.com/paper/2110.01823/) (2021)
- [Sparse Black-box Video Attack with Reinforcement Learning](https://mlsanity.com/paper/2001.03754/) (2020)
- [Boosting the Transferability of Video Adversarial Examples via Temporal Translation](https://mlsanity.com/paper/2110.09075/) (2021)
- [Motion-Excited Sampler: Video Adversarial Attack with Sparked Prior](https://mlsanity.com/paper/2003.07637/) (2020)

## Built on by (most cited first, 21 of 21)

- [AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception](https://mlsanity.com/paper/2307.13933/) (2023)
- [SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation](https://mlsanity.com/paper/2312.04913/) (2023)
- [CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness](https://mlsanity.com/paper/2502.14914/) (2025)
- [DUAW: Data-free Universal Adversarial Watermark against Stable Diffusion Customization](https://mlsanity.com/paper/2308.09889/) (2023)
- [DSRC: Learning Density-insensitive and Semantic-aware Collaborative Representation against Corruptions](https://mlsanity.com/paper/2412.10739/) (2024)
- [Fast Adversarial Training with Weak-to-Strong Spatial-Temporal Consistency in the Frequency Domain on Videos](https://mlsanity.com/paper/2504.14921/) (2025)
- [LogoStyleFool: Vitiating Video Recognition Systems via Logo Style Transfer](https://mlsanity.com/paper/2312.09935/) (2023)
- [Ask, Attend, Attack: A Effective Decision-Based Black-Box Targeted Attack for Image-to-Text Models](https://mlsanity.com/paper/2408.08989/) (2024)
- [VideoPure: Diffusion-based Adversarial Purification for Video Recognition](https://mlsanity.com/paper/2501.14999/) (2025)
- [ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinement](https://mlsanity.com/paper/2512.13303/) (2025)
- [VLA-Hijack: A Transferable Patch Attack against Vision-Language-Action Models via Visual Proprioception Hijacking](https://mlsanity.com/paper/2605.28083/) (2026)
- [Query-Efficient Video Adversarial Attack with Stylized Logo on Service Computing](https://mlsanity.com/paper/2408.12099/) (2024)
- [Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment](https://mlsanity.com/paper/2506.01511/) (2025)
- [LocalStyleFool: Regional Video Style Transfer Attack Using Segment Anything Model](https://mlsanity.com/paper/2403.11656/) (2024)
- [Suppressing Uncertainties in Degradation Estimation for Blind Super-Resolution](https://mlsanity.com/paper/2406.16459/) (2024)
- [TagOOD: A Novel Approach to Out-of-Distribution Detection via Vision-Language Representations and Class Center Learning](https://mlsanity.com/paper/2408.15566/) (2024)
- [Adaptive Multi-modal Fusion of Spatially Variant Kernel Refinement with Diffusion Model for Blind Image Super-Resolution](https://mlsanity.com/paper/2403.05808/) (2024)
- [SVASTIN: Sparse Video Adversarial Attack via Spatio-Temporal Invertible Neural Networks](https://mlsanity.com/paper/2406.01894/) (2024)
- [Dynamic Semantic-Aware Correlation Modeling for UAV Tracking](https://mlsanity.com/paper/2510.21351/) (2025)
- [Exploring the Adversarial Robustness of Face Forgery Detection with Decision-based Black-box Attacks](https://mlsanity.com/paper/2310.12017/) (2023)
- [Improving Adversarial Transferability with Neighbourhood Gradient Information](https://mlsanity.com/paper/2408.05745/) (2024)

## Abstract (arXiv)

Although Deep Neural Networks (DNNs) have demonstrated excellent performance, they are vulnerable to adversarial patches that introduce perceptible and localized perturbations to the input. Generating adversarial patches on images has received much attention, while adversarial patches on videos have not been well investigated. Further, decision-based attacks, where attackers only access the predicted hard labels by querying threat models, have not been well explored on video models either, even if they are practical in real-world video recognition scenes. The absence of such studies leads to a huge gap in the robustness assessment for video models. To bridge this gap, this work first explores decision-based patch attacks on video models. We analyze that the huge parameter space brought by videos and the minimal information returned by decision-based models both greatly increase the attack difficulty and query burden. To achieve a query-efficient attack, we propose a spatial-temporal differential evolution (STDE) framework. First, STDE introduces target videos as patch textures and only adds patches on keyframes that are adaptively selected by temporal difference. Second, STDE takes minimizing the patch area as the optimization objective and adopts spatialtemporal mutation and crossover to search for the global optimum without falling into the local optimum. Experiments show STDE has demonstrated state-of-the-art performance in terms of threat, efficiency and imperceptibility. Hence, STDE has the potential to be a powerful tool for evaluating the robustness of video recognition models.