# Sparse Black-box Video Attack with Reinforcement Learning

arXiv: https://arxiv.org/abs/2001.03754 · HTML: https://mlsanity.com/paper/2001.03754/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2020.

## Built on by (most cited first, 24 of 24)

- [Towards Transferable Adversarial Attacks on Vision Transformers](https://mlsanity.com/paper/2109.04176/) (2021)
- [Adversarial Attacks on Black Box Video Classifiers: Leveraging the Power of Geometric Transformations](https://mlsanity.com/paper/2110.01823/) (2021)
- [Unified Adversarial Patch for Cross-modal Attacks in the Physical World](https://mlsanity.com/paper/2307.07859/) (2023)
- [Efficient Decision-based Black-box Patch Attacks on Video Recognition](https://mlsanity.com/paper/2303.11917/) (2023)
- [Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos](https://mlsanity.com/paper/2301.00896/) (2023)
- [GAMA: Generative Adversarial Multi-Object Scene Attacks](https://mlsanity.com/paper/2209.09502/) (2022)
- [Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation](https://mlsanity.com/paper/2306.16170/) (2023)
- [Reinforcement Learning Based Sparse Black-box Adversarial Attack on Video Recognition Models](https://mlsanity.com/paper/2108.13872/) (2021)
- [Sparse Adversarial Video Attacks with Spatial Transformations](https://mlsanity.com/paper/2111.05468/) (2021)
- [Visual Adversarial Attacks and Defenses in the Physical World: A Survey](https://mlsanity.com/paper/2211.01671/) (2022)
- [Adversarial Machine Learning Attacks Against Video Anomaly Detection Systems](https://mlsanity.com/paper/2204.03141/) (2022)
- [Cross-Modal Transferable Image-to-Video Attack on Video Quality Metrics](https://mlsanity.com/paper/2501.08415/) (2025)
- [Understanding the Vulnerability of Skeleton-based Human Activity Recognition via Black-box Attack](https://mlsanity.com/paper/2211.11312/) (2022)
- [Improving Viewpoint Robustness for Visual Recognition via Adversarial Training](https://mlsanity.com/paper/2307.11528/) (2023)
- [LogoStyleFool: Vitiating Video Recognition Systems via Logo Style Transfer](https://mlsanity.com/paper/2312.09935/) (2023)
- [A Comprehensive Study on the Robustness of Image Classification and Object Detection in Remote Sensing: Surveying and Benchmarking](https://mlsanity.com/paper/2306.12111/) (2023)
- [Multimodal Attack Detection for Action Recognition Models](https://mlsanity.com/paper/2404.10790/) (2024)
- [Revisiting the Trade-off between Accuracy and Robustness via Weight Distribution of Filters](https://mlsanity.com/paper/2306.03430/) (2023)
- [Query-Efficient Video Adversarial Attack with Stylized Logo on Service Computing](https://mlsanity.com/paper/2408.12099/) (2024)
- [LocalStyleFool: Regional Video Style Transfer Attack Using Segment Anything Model](https://mlsanity.com/paper/2403.11656/) (2024)
- [TenAd: A Tensor-based Low-rank Black Box Adversarial Attack for Video Classification](https://mlsanity.com/paper/2504.01228/) (2025)
- [A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability](https://mlsanity.com/paper/2505.10351/) (2025)
- [Cross-Modal Transferable Adversarial Attacks from Images to Videos](https://mlsanity.com/paper/2112.05379/) (2021)
- [Robustness Evaluation for Video Models with Reinforcement Learning](https://mlsanity.com/paper/2506.05431/) (2025)

## Abstract (arXiv)

Adversarial attacks on video recognition models have been explored recently. However, most existing works treat each video frame equally and ignore their temporal interactions. To overcome this drawback, a few methods try to select some key frames and then perform attacks based on them. Unfortunately, their selection strategy is independent of the attacking step, therefore the resulting performance is limited. Instead, we argue the frame selection phase is closely relevant with the attacking phase. The key frames should be adjusted according to the attacking results. For that, we formulate the black-box video attacks into a Reinforcement Learning (RL) framework. Specifically, the environment in RL is set as the recognition model, and the agent in RL plays the role of frame selecting. By continuously querying the recognition models and receiving the attacking feedback, the agent gradually adjusts its frame selection strategy and adversarial perturbations become smaller and smaller. We conduct a series of experiments with two mainstream video recognition models: C3D and LRCN on the public UCF-101 and HMDB-51 datasets. The results demonstrate that the proposed method can significantly reduce the adversarial perturbations with efficient query times.