# Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos

arXiv: https://arxiv.org/abs/2301.00896 · HTML: https://mlsanity.com/paper/2301.00896/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 57 references that are in the map.
- First posted on arXiv 3 Jan 2023.

## Builds on (most cited first, 30 of 36)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Black-box Adversarial Attacks with Limited Queries and Information](https://mlsanity.com/paper/1804.08598/) (2018)
- [Adversarial Examples: Attacks and Defenses for Deep Learning](https://mlsanity.com/paper/1712.07107/) (2017)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Salient Object Detection in the Deep Learning Era: An In-Depth Survey](https://mlsanity.com/paper/1904.09146/) (2019)
- [Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priors](https://mlsanity.com/paper/1807.07978/) (2018)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Robustness of Graph Neural Networks at Scale](https://mlsanity.com/paper/2110.14038/) (2021)
- [Adversarial Sticker: A Stealthy Attack Method in the Physical World](https://mlsanity.com/paper/2104.06728/) (2021)
- [A Comprehensive Study of Deep Video Action Recognition](https://mlsanity.com/paper/2012.06567/) (2020)
- [RobustART: Benchmarking Robustness on Architecture Design and Training Techniques](https://mlsanity.com/paper/2109.05211/) (2021)
- [Black-box Adversarial Attacks on Video Recognition Models](https://mlsanity.com/paper/1904.05181/) (2019)
- [Sparse Adversarial Perturbations for Videos](https://mlsanity.com/paper/1803.02536/) (2018)
- [Query-Efficient Black-box Adversarial Attacks Guided by a Transfer-based Prior](https://mlsanity.com/paper/2203.06560/) (2022)
- [Heuristic Black-box Adversarial Attacks on Video Recognition Models](https://mlsanity.com/paper/1911.09449/) (2019)
- [Adversarial Perturbations Against Real-Time Video Classification Systems](https://mlsanity.com/paper/1807.00458/) (2018)
- [Adversarial Attacks on Black Box Video Classifiers: Leveraging the Power of Geometric Transformations](https://mlsanity.com/paper/2110.01823/) (2021)
- [Sparse Black-box Video Attack with Reinforcement Learning](https://mlsanity.com/paper/2001.03754/) (2020)

## Built on by (most cited first, 20 of 20)

- [Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models](https://mlsanity.com/paper/2508.12880/) (2025)
- [From Zero to Detail: Deconstructing Ultra-High-Definition Image Restoration from Progressive Spectral Perspective](https://mlsanity.com/paper/2503.13165/) (2025)
- [LUVE : Latent-Cascaded Ultra-High-Resolution Video Generation with Dual Frequency Experts](https://mlsanity.com/paper/2602.11564/) (2026)
- [Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation](https://mlsanity.com/paper/2306.16170/) (2023)
- [Fast Adversarial Training with Weak-to-Strong Spatial-Temporal Consistency in the Frequency Domain on Videos](https://mlsanity.com/paper/2504.14921/) (2025)
- [Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation](https://mlsanity.com/paper/2312.05508/) (2023)
- [RunawayEvil: Jailbreaking the Image-to-Video Generative Models](https://mlsanity.com/paper/2512.06674/) (2025)
- [LogoStyleFool: Vitiating Video Recognition Systems via Logo Style Transfer](https://mlsanity.com/paper/2312.09935/) (2023)
- [Sequential Manipulation Against Rank Aggregation: Theory and Algorithm](https://mlsanity.com/paper/2407.01916/) (2024)
- [From Zero to Detail: A Progressive Spectral Decoupling Paradigm for UHD Image Restoration with New Benchmark](https://mlsanity.com/paper/2604.15654/) (2026)
- [A Comprehensive Study on the Robustness of Image Classification and Object Detection in Remote Sensing: Surveying and Benchmarking](https://mlsanity.com/paper/2306.12111/) (2023)
- [Revisiting the Trade-off between Accuracy and Robustness via Weight Distribution of Filters](https://mlsanity.com/paper/2306.03430/) (2023)
- [Query-Efficient Video Adversarial Attack with Stylized Logo on Service Computing](https://mlsanity.com/paper/2408.12099/) (2024)
- [Attack Anything: Blind DNNs via Universal Background Adversarial Attack](https://mlsanity.com/paper/2409.00029/) (2024)
- [FABLE: A Localized, Targeted Adversarial Attack on Weather Forecasting Models](https://mlsanity.com/paper/2505.12167/) (2025)
- [An Effective End-to-End Solution for Multimodal Action Recognition](https://mlsanity.com/paper/2506.09345/) (2025)
- [Exposing and Defending the Achilles' Heel of Video Mixture-of-Experts](https://mlsanity.com/paper/2602.01369/) (2026)
- [LocalStyleFool: Regional Video Style Transfer Attack Using Segment Anything Model](https://mlsanity.com/paper/2403.11656/) (2024)
- [Robustness Evaluation for Video Models with Reinforcement Learning](https://mlsanity.com/paper/2506.05431/) (2025)
- [Towards Class-wise Fair Adversarial Training via Anti-Bias Soft Label Distillation](https://mlsanity.com/paper/2506.08611/) (2025)

## Abstract (arXiv)

Adversarial robustness assessment for video recognition models has raised concerns owing to their wide applications on safety-critical tasks. Compared with images, videos have much high dimension, which brings huge computational costs when generating adversarial videos. This is especially serious for the query-based black-box attacks where gradient estimation for the threat models is usually utilized, and high dimensions will lead to a large number of queries. To mitigate this issue, we propose to simultaneously eliminate the temporal and spatial redundancy within the video to achieve an effective and efficient gradient estimation on the reduced searching space, and thus query number could decrease. To implement this idea, we design the novel Adversarial spatial-temporal Focus (AstFocus) attack on videos, which performs attacks on the simultaneously focused key frames and key regions from the inter-frames and intra-frames in the video. AstFocus attack is based on the cooperative Multi-Agent Reinforcement Learning (MARL) framework. One agent is responsible for selecting key frames, and another agent is responsible for selecting key regions. These two agents are jointly trained by the common rewards received from the black-box threat models to perform a cooperative prediction. By continuously querying, the reduced searching space composed of key frames and key regions is becoming precise, and the whole query number becomes less than that on the original video. Extensive experiments on four mainstream video recognition models and three widely used action recognition datasets demonstrate that the proposed AstFocus attack outperforms the SOTA methods, which is prevenient in fooling rate, query number, time, and perturbation magnitude at the same.