# Multimodal Attack Detection for Action Recognition Models

arXiv: https://arxiv.org/abs/2404.10790 · HTML: https://mlsanity.com/paper/2404.10790/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 13 Apr 2024.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Black-box Adversarial Attacks with Limited Queries and Information](https://mlsanity.com/paper/1804.08598/) (2018)
- [Mitigating Adversarial Effects Through Randomization](https://mlsanity.com/paper/1711.01991/) (2017)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022)
- [Adversarial Risk and the Dangers of Evaluating Against Weak Attacks](https://mlsanity.com/paper/1802.05666/) (2018)
- [A study of the effect of JPG compression on adversarial images](https://mlsanity.com/paper/1608.00853/) (2016)
- [On Physical Adversarial Patches for Object Detection](https://mlsanity.com/paper/1906.11897/) (2019)
- [Towards Transferable Adversarial Attacks on Vision Transformers](https://mlsanity.com/paper/2109.04176/) (2021)
- [Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models](https://mlsanity.com/paper/2301.00182/) (2022)
- [Exploring Target Representations for Masked Autoencoders](https://mlsanity.com/paper/2209.03917/) (2022)
- [Heuristic Black-box Adversarial Attacks on Video Recognition Models](https://mlsanity.com/paper/1911.09449/) (2019)
- [Over-the-Air Adversarial Flickering Attacks against Video Recognition Networks](https://mlsanity.com/paper/2002.05123/) (2020)
- [Adversarial Attacks on Black Box Video Classifiers: Leveraging the Power of Geometric Transformations](https://mlsanity.com/paper/2110.01823/) (2021)
- [ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches](https://mlsanity.com/paper/2203.04412/) (2022)
- [Delving into CLIP latent space for Video Anomaly Recognition](https://mlsanity.com/paper/2310.02835/) (2023)
- [Sparse Black-box Video Attack with Reinforcement Learning](https://mlsanity.com/paper/2001.03754/) (2020)
- [Pixle: a fast and effective black-box attack based on rearranging pixels](https://mlsanity.com/paper/2202.02236/) (2022)
- [Just One Moment: Structural Vulnerability of Deep Action Recognition against One Frame Attack](https://mlsanity.com/paper/2011.14585/) (2020)
- [Adversarial Machine Learning Attacks Against Video Anomaly Detection Systems](https://mlsanity.com/paper/2204.03141/) (2022)
- [Temporal Shuffling for Defending Deep Action Recognition Models against Adversarial Attacks](https://mlsanity.com/paper/2112.07921/) (2021)

## Built on by (most cited first, 5 of 5)

- [A Neurosymbolic Framework for Interpretable Cognitive Attack Detection in Augmented Reality](https://mlsanity.com/paper/2508.09185/) (2025)
- [Detecting Adversarial Examples](https://mlsanity.com/paper/2410.17442/) (2024)
- [Universal and Efficient Detection of Adversarial Data through Nonuniform Impact on Network Layers](https://mlsanity.com/paper/2506.20816/) (2025)
- [Concept-SAE: A Controllable and Invertible Concept Interface for Sparse Autoencoders](https://mlsanity.com/paper/2509.22015/) (2025)
- [Detecting Adversarial Data via Provable Adversarial Noise Amplification](https://mlsanity.com/paper/2605.02109/) (2026)

## Abstract (arXiv)

Adversarial machine learning attacks on video action recognition models is a growing research area and many effective attacks were introduced in recent years. These attacks show that action recognition models can be breached in many ways. Hence using these models in practice raises significant security concerns. However, there are very few works which focus on defending against or detecting attacks. In this work, we propose a novel universal detection method which is compatible with any action recognition model. In our extensive experiments, we show that our method consistently detects various attacks against different target models with high true positive rates while satisfying very low false positive rates. Tested against four state-of-the-art attacks targeting four action recognition models, the proposed detector achieves an average AUC of 0.911 over 16 test cases while the best performance achieved by the existing detectors is 0.645 average AUC. This 41.2% improvement is enabled by the robustness of the proposed detector to varying attack methods and target models. The lowest AUC achieved by our detector across the 16 test cases is 0.837 while the competing detector's performance drops as low as 0.211. We also show that the proposed detector is robust to varying attack strengths. In addition, we analyze our method's real-time performance with different hardware setups to demonstrate its potential as a practical defense mechanism.