# Reinforcement Learning Based Sparse Black-box Adversarial Attack on Video Recognition Models

arXiv: https://arxiv.org/abs/2108.13872 · HTML: https://mlsanity.com/paper/2108.13872/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 29 Aug 2021.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Tactics of Adversarial Attack on Deep Reinforcement Learning Agents](https://mlsanity.com/paper/1703.06748/) (2017)
- [Query-Efficient Hard-label Black-box Attack:An Optimization-based Approach](https://mlsanity.com/paper/1807.04457/) (2018)
- [Sign-OPT: A Query-Efficient Hard-label Adversarial Attack](https://mlsanity.com/paper/1909.10773/) (2019)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [PatchAttack: A Black-box Texture-based Attack with Reinforcement Learning](https://mlsanity.com/paper/2004.05682/) (2020)
- [Black-box Adversarial Attacks on Video Recognition Models](https://mlsanity.com/paper/1904.05181/) (2019)
- [Sparse Adversarial Perturbations for Videos](https://mlsanity.com/paper/1803.02536/) (2018)
- [Adversarial Perturbations Against Real-Time Video Classification Systems](https://mlsanity.com/paper/1807.00458/) (2018)
- [Sparse Black-box Video Attack with Reinforcement Learning](https://mlsanity.com/paper/2001.03754/) (2020)
- [Adversarial Framing for Image and Video Classification](https://mlsanity.com/paper/1812.04599/) (2018)
- [Identifying and Resisting Adversarial Videos Using Temporal Consistency](https://mlsanity.com/paper/1909.04837/) (2019)

## Built on by (most cited first, 12 of 12)

- [Efficient Decision-based Black-box Patch Attacks on Video Recognition](https://mlsanity.com/paper/2303.11917/) (2023)
- [StyleFool: Fooling Video Classification Systems via Style Transfer](https://mlsanity.com/paper/2203.16000/) (2022)
- [Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos](https://mlsanity.com/paper/2301.00896/) (2023)
- [Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Models](https://mlsanity.com/paper/2406.07145/) (2024)
- [LogoStyleFool: Vitiating Video Recognition Systems via Logo Style Transfer](https://mlsanity.com/paper/2312.09935/) (2023)
- [Query-Efficient Video Adversarial Attack with Stylized Logo on Service Computing](https://mlsanity.com/paper/2408.12099/) (2024)
- [LocalStyleFool: Regional Video Style Transfer Attack Using Segment Anything Model](https://mlsanity.com/paper/2403.11656/) (2024)
- [When Measures are Unreliable: Imperceptible Adversarial Perturbations toward Top-k Multi-Label Learning](https://mlsanity.com/paper/2309.00007/) (2023)
- [LLM-Assisted Red Teaming of Diffusion Models through "Failures Are Fated, But Can Be Faded"](https://mlsanity.com/paper/2410.16738/) (2024)
- [Adversarial Attacks in Multimodal Systems: A Practitioner's Survey](https://mlsanity.com/paper/2505.03084/) (2025)
- [Robustness Evaluation for Video Models with Reinforcement Learning](https://mlsanity.com/paper/2506.05431/) (2025)
- [FeatureFool: Zero-Query Fooling of Video Models via Feature Map](https://mlsanity.com/paper/2510.18362/) (2025)

## Abstract (arXiv)

We explore the black-box adversarial attack on video recognition models. Attacks are only performed on selected key regions and key frames to reduce the high computation cost of searching adversarial perturbations on a video due to its high dimensionality. To select key frames, one way is to use heuristic algorithms to evaluate the importance of each frame and choose the essential ones. However, it is time inefficient on sorting and searching. In order to speed up the attack process, we propose a reinforcement learning based frame selection strategy. Specifically, the agent explores the difference between the original class and the target class of videos to make selection decisions. It receives rewards from threat models which indicate the quality of the decisions. Besides, we also use saliency detection to select key regions and only estimate the sign of gradient instead of the gradient itself in zeroth order optimization to further boost the attack process. We can use the trained model directly in the untargeted attack or with little fine-tune in the targeted attack, which saves computation time. A range of empirical results on real datasets demonstrate the effectiveness and efficiency of the proposed method.