# Is Appearance Free Action Recognition Possible?

arXiv: https://arxiv.org/abs/2207.06261 · HTML: https://mlsanity.com/paper/2207.06261/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 64 references that are in the map.
- First posted on arXiv 13 Jul 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition](https://mlsanity.com/paper/1912.05534/) (2019)
- [A Comprehensive Study of Deep Video Action Recognition](https://mlsanity.com/paper/2012.06567/) (2020)
- [It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos](https://mlsanity.com/paper/1604.00136/) (2016)
- [Only Time Can Tell: Discovering Temporal Data for Temporal Modeling](https://mlsanity.com/paper/1907.08340/) (2019)
- [Betrayed by Motion: Camouflaged Object Discovery via Motion Segmentation](https://mlsanity.com/paper/2011.11630/) (2020)
- [On the Integration of Optical Flow and Action Recognition](https://mlsanity.com/paper/1712.08416/) (2017)
- [A Deeper Dive Into What Deep Spatiotemporal Networks Encode: Quantifying Static vs. Dynamic Information](https://mlsanity.com/paper/2206.02846/) (2022)
- [Video Time: Properties, Encoders and Evaluation](https://mlsanity.com/paper/1807.06980/) (2018)
- [Human Action Recognition without Human](https://mlsanity.com/paper/1608.07876/) (2016)
- [Review of Action Recognition and Detection Methods](https://mlsanity.com/paper/1610.06906/) (2016)
- [Interpreting video features: a comparison of 3D convolutional networks and convolutional LSTM networks](https://mlsanity.com/paper/2002.00367/) (2020)

## Built on by (most cited first, 16 of 16)

- [Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foreground](https://mlsanity.com/paper/2211.12883/) (2022)
- [Understanding Video Transformers via Universal Concept Discovery](https://mlsanity.com/paper/2401.10831/) (2024)
- [Prisma: An Open Source Toolkit for Mechanistic Interpretability in Vision and Video](https://mlsanity.com/paper/2504.19475/) (2025)
- [Selective, Interpretable, and Motion Consistent Privacy Attribute Obfuscation for Action Recognition](https://mlsanity.com/paper/2403.12710/) (2024)
- [Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networks](https://mlsanity.com/paper/2211.01783/) (2022)
- [Detection Defenses: An Empty Promise against Adversarial Patch Attacks on Optical Flow](https://mlsanity.com/paper/2310.17403/) (2023)
- [Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception](https://mlsanity.com/paper/2405.16493/) (2024)
- [The PanAf-FGBG Dataset: Understanding the Impact of Backgrounds in Wildlife Behaviour Recognition](https://mlsanity.com/paper/2502.21201/) (2025)
- [Video-DPRP: A Differentially Private Approach for Visual Privacy-Preserving Video Human Activity Recognition](https://mlsanity.com/paper/2503.02132/) (2025)
- [Adaptive occlusion sensitivity analysis for visually explaining video recognition networks](https://mlsanity.com/paper/2207.12859/) (2022)
- [Better, But Not Sufficient: Testing Video ANNs Against Macaque IT Dynamics](https://mlsanity.com/paper/2601.03392/) (2026)
- [Identifying Ethical Biases in Action Recognition Models](https://mlsanity.com/paper/2604.17971/) (2026)
- [TrAction: Action Recognition with Sparse Trajectories](https://mlsanity.com/paper/2606.03490/) (2026)
- [Self-Supervised Video Representation Learning in a Heuristic Decoupled Perspective](https://mlsanity.com/paper/2407.14069/) (2024)
- [MoExDA: Domain Adaptation for Edge-based Action Recognition](https://mlsanity.com/paper/2508.02981/) (2025)
- [Appearance-free Action Recognition: Zero-shot Generalization in Humans and a Two-Pathway Model](https://mlsanity.com/paper/2604.16675/) (2026)

## Abstract (arXiv)

Intuition might suggest that motion and dynamic information are key to video-based action recognition. In contrast, there is evidence that state-of-the-art deep-learning video understanding architectures are biased toward static information available in single frames. Presently, a methodology and corresponding dataset to isolate the effects of dynamic information in video are missing. Their absence makes it difficult to understand how well contemporary architectures capitalize on dynamic vs. static information. We respond with a novel Appearance Free Dataset (AFD) for action recognition. AFD is devoid of static information relevant to action recognition in a single frame. Modeling of the dynamics is necessary for solving the task, as the action is only apparent through consideration of the temporal dimension. We evaluated 11 contemporary action recognition architectures on AFD as well as its related RGB video. Our results show a notable decrease in performance for all architectures on AFD compared to RGB. We also conducted a complimentary study with humans that shows their recognition accuracy on AFD and RGB is very similar and much better than the evaluated architectures on AFD. Our results motivate a novel architecture that revives explicit recovery of optical flow, within a contemporary design for best performance on AFD and RGB.