# Temporal Shuffling for Defending Deep Action Recognition Models against Adversarial Attacks

arXiv: https://arxiv.org/abs/2112.07921 · HTML: https://mlsanity.com/paper/2112.07921/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 15 Dec 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Mitigating Adversarial Effects Through Randomization](https://mlsanity.com/paper/1711.01991/) (2017)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks](https://mlsanity.com/paper/1905.13545/) (2019)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [A study of the effect of JPG compression on adversarial images](https://mlsanity.com/paper/1608.00853/) (2016)
- [Towards Robust Neural Networks via Random Self-ensemble](https://mlsanity.com/paper/1712.00673/) (2017)
- [Sparse Adversarial Perturbations for Videos](https://mlsanity.com/paper/1803.02536/) (2018)
- [Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classification](https://mlsanity.com/paper/1709.05583/) (2017)
- [Not Using the Car to See the Sidewalk: Quantifying and Controlling the Effects of Context in Classification and Segmentation](https://mlsanity.com/paper/1812.06707/) (2018)
- [Object Recognition with and without Objects](https://mlsanity.com/paper/1611.06596/) (2016)
- [Over-the-Air Adversarial Flickering Attacks against Video Recognition Networks](https://mlsanity.com/paper/2002.05123/) (2020)
- [Adversarial Perturbations Against Real-Time Video Classification Systems](https://mlsanity.com/paper/1807.00458/) (2018)
- [Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition](https://mlsanity.com/paper/2107.00606/) (2021)
- [Defending Against Multiple and Unforeseen Adversarial Videos](https://mlsanity.com/paper/2009.05244/) (2020)

## Built on by (most cited first, 7 of 7)

- [Multimodal Attack Detection for Action Recognition Models](https://mlsanity.com/paper/2404.10790/) (2024)
- [Query-Efficient Video Adversarial Attack with Stylized Logo on Service Computing](https://mlsanity.com/paper/2408.12099/) (2024)
- [Detecting Adversarial Examples](https://mlsanity.com/paper/2410.17442/) (2024)
- [Universal and Efficient Detection of Adversarial Data through Nonuniform Impact on Network Layers](https://mlsanity.com/paper/2506.20816/) (2025)
- [Adversarial Video Promotion Against Text-to-Video Retrieval](https://mlsanity.com/paper/2508.06964/) (2025)
- [FeatureFool: Zero-Query Fooling of Video Models via Feature Map](https://mlsanity.com/paper/2510.18362/) (2025)
- [FMVP: Masked Flow Matching for Adversarial Video Purification](https://mlsanity.com/paper/2601.02228/) (2026)

## Abstract (arXiv)

Recently, video-based action recognition methods using convolutional neural networks (CNNs) achieve remarkable recognition performance. However, there is still lack of understanding about the generalization mechanism of action recognition models. In this paper, we suggest that action recognition models rely on the motion information less than expected, and thus they are robust to randomization of frame orders. Furthermore, we find that motion monotonicity remaining after randomization also contributes to such robustness. Based on this observation, we develop a novel defense method using temporal shuffling of input videos against adversarial attacks for action recognition models. Another observation enabling our defense method is that adversarial perturbations on videos are sensitive to temporal destruction. To the best of our knowledge, this is the first attempt to design a defense method without additional training for 3D CNN-based video action recognition models.