# Large-scale Robustness Analysis of Video Action Recognition Models

arXiv: https://arxiv.org/abs/2207.01398 · HTML: https://mlsanity.com/paper/2207.01398/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 73 references that are in the map.
- First posted on arXiv 4 Jul 2022.

## Builds on (most cited first, 30 of 51)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [A Fourier Perspective on Model Robustness in Computer Vision](https://mlsanity.com/paper/1906.08988/) (2019)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)

## Built on by (most cited first, 29 of 29)

- [Video Test-Time Adaptation for Action Recognition](https://mlsanity.com/paper/2211.15393/) (2022)
- [MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning](https://mlsanity.com/paper/2406.14796/) (2024)
- [EZ-CLIP: Efficient Zeroshot Video Action Recognition](https://mlsanity.com/paper/2312.08010/) (2023)
- [Robustness Analysis on Foundational Segmentation Models](https://mlsanity.com/paper/2306.09278/) (2023)
- [Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions](https://mlsanity.com/paper/2403.20254/) (2024)
- [EventTransAct: A video transformer-based framework for Event-camera based action recognition](https://mlsanity.com/paper/2308.13711/) (2023)
- [No More Shortcuts: Realizing the Potential of Temporal Self-Supervision](https://mlsanity.com/paper/2312.13008/) (2023)
- [Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness](https://mlsanity.com/paper/2410.03030/) (2024)
- [A Large-Scale Analysis on Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2306.06010/) (2023)
- [Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception](https://mlsanity.com/paper/2405.16493/) (2024)
- [Uncovering the Hidden Dynamics of Video Self-supervised Learning under Distribution Shifts](https://mlsanity.com/paper/2306.02014/) (2023)
- [Towards Synthetic Data Generation for Improved Pain Recognition in Videos under Patient Constraints](https://mlsanity.com/paper/2409.16382/) (2024)
- [Learning Video Representations without Natural Videos](https://mlsanity.com/paper/2410.24213/) (2024)
- [A Large-Scale Analysis on Contextual Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2504.06153/) (2025)
- [Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation](https://mlsanity.com/paper/2506.12481/) (2025)
- [RobustFormer: Noise-Robust Pre-training for images and videos](https://mlsanity.com/paper/2411.13040/) (2024)
- [textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time](https://mlsanity.com/paper/2506.00358/) (2025)
- [Adversarial Augmentation Training Makes Action Recognition Models More Robust to Realistic Video Distribution Shifts](https://mlsanity.com/paper/2401.11406/) (2024)
- [No Time to Waste: Squeeze Time into Channel for Mobile Video Understanding](https://mlsanity.com/paper/2405.08344/) (2024)
- [Punching Bag vs. Punching Person: Motion Transferability in Videos](https://mlsanity.com/paper/2508.00085/) (2025)
- [RO-Bench: Large-scale robustness evaluation of MLLMs with text-driven counterfactual videos](https://mlsanity.com/paper/2510.08936/) (2025)
- [PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs](https://mlsanity.com/paper/2511.10979/) (2025)
- [Identifying Ethical Biases in Action Recognition Models](https://mlsanity.com/paper/2604.17971/) (2026)
- [High-Order Evolving Graphs for Enhanced Representation of Traffic Dynamics](https://mlsanity.com/paper/2409.11206/) (2024)
- [RobustGait: Robustness Analysis for Appearance Based Gait Recognition](https://mlsanity.com/paper/2511.13065/) (2025)
- [Effect of Activation Function and Model Optimizer on the Performance of Human Activity Recognition System Using Various Deep Learning Models](https://mlsanity.com/paper/2512.20104/) (2025)
- [Spike-NVPT: Learning Robust Visual Prompts via Bio-Inspired Temporal Filtering and Discretization](https://mlsanity.com/paper/2604.18284/) (2026)
- [Can We Trust Video Hallucination Detectors? VidHalLoc for Evaluating the Evaluators](https://mlsanity.com/paper/2609.09895/) (2026)
- [CE4L: Continual Ego, Exo, and Ego-Exo Learning](https://mlsanity.com/paper/2609.23492/) (2026)

## Abstract (arXiv)

We have seen a great progress in video action recognition in recent years. There are several models based on convolutional neural network (CNN) and some recent transformer based approaches which provide top performance on existing benchmarks. In this work, we perform a large-scale robustness analysis of these existing models for video action recognition. We focus on robustness against real-world distribution shift perturbations instead of adversarial perturbations. We propose four different benchmark datasets, HMDB51-P, UCF101-P, Kinetics400-P, and SSv2-P to perform this analysis. We study robustness of six state-of-the-art action recognition models against 90 different perturbations. The study reveals some interesting findings, 1) transformer based models are consistently more robust compared to CNN based models, 2) Pretraining improves robustness for Transformer based models more than CNN based models, and 3) All of the studied models are robust to temporal perturbations for all datasets but SSv2; suggesting the importance of temporal information for action recognition varies based on the dataset and activities. Next, we study the role of augmentations in model robustness and present a real-world dataset, UCF101-DS, which contains realistic distribution shifts, to further validate some of these findings. We believe this study will serve as a benchmark for future research in robust video action recognition.