# Deep Analysis of CNN-based Spatio-temporal Representations for Action Recognition

arXiv: https://arxiv.org/abs/2010.11757 · HTML: https://mlsanity.com/paper/2010.11757/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 30 of 45)

- [FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2203.09463/) (2022)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [Semi-Supervised Action Recognition with Temporal Contrastive Learning](https://mlsanity.com/paper/2102.02751/) (2021)
- [Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing](https://mlsanity.com/paper/2110.15128/) (2021)
- [MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge](https://mlsanity.com/paper/2303.08914/) (2023)
- [CLIP-guided Prototype Modulating for Few-shot Action Recognition](https://mlsanity.com/paper/2303.02982/) (2023)
- [AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition](https://mlsanity.com/paper/2105.05165/) (2021)
- [Dynamic Network Quantization for Efficient Video Inference](https://mlsanity.com/paper/2108.10394/) (2021)
- [Benchmarking the Robustness of Spatial-Temporal Models Against Corruptions](https://mlsanity.com/paper/2110.06513/) (2021)
- [DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition](https://mlsanity.com/paper/2203.10233/) (2022)
- [Large-scale Robustness Analysis of Video Action Recognition Models](https://mlsanity.com/paper/2207.01398/) (2022)
- [Can An Image Classifier Suffice For Action Recognition?](https://mlsanity.com/paper/2106.14104/) (2021)
- [On Modality Bias Recognition and Reduction](https://mlsanity.com/paper/2202.12690/) (2022)
- [Temporal Coherent Test-Time Optimization for Robust Video Classification](https://mlsanity.com/paper/2302.14309/) (2023)
- [Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognition](https://mlsanity.com/paper/2412.11228/) (2024)
- [Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition](https://mlsanity.com/paper/2110.15639/) (2021)
- [Recur, Attend or Convolve? On Whether Temporal Modeling Matters for Cross-Domain Robustness in Action Recognition](https://mlsanity.com/paper/2112.12175/) (2021)
- [A Dynamic Graph CNN with Cross-Representation Distillation for Event-Based Recognition](https://mlsanity.com/paper/2302.04177/) (2023)
- [Video Recognition in Portrait Mode](https://mlsanity.com/paper/2312.13746/) (2023)
- [Action Selection Learning for Multi-label Multi-view Action Recognition](https://mlsanity.com/paper/2410.03302/) (2024)
- [Mixture of Experts Guided by Gaussian Splatters Matters: A new Approach to Weakly-Supervised Video Anomaly Detection](https://mlsanity.com/paper/2508.06318/) (2025)
- [Spatio-Temporal Fluid Dynamics Modeling via Physical-Awareness and Parameter Diffusion Guidance](https://mlsanity.com/paper/2403.13850/) (2024)
- [Fine-grained Activities of People Worldwide](https://mlsanity.com/paper/2207.05182/) (2022)
- [SVFormer: A Direct Training Spiking Transformer for Efficient Video Action Recognition](https://mlsanity.com/paper/2406.15034/) (2024)
- [M2A: Motion Aware Attention for Accurate Video Action Recognition](https://mlsanity.com/paper/2111.09976/) (2021)
- [Should I take a walk? Estimating Energy Expenditure from Video Data](https://mlsanity.com/paper/2202.00712/) (2022)
- [Explore Spatio-temporal Aggregation for Insubstantial Object Detection: Benchmark Dataset and Baseline](https://mlsanity.com/paper/2206.11459/) (2022)
- [On the Surprising Effectiveness of Transformers in Low-Labeled Video Recognition](https://mlsanity.com/paper/2209.07474/) (2022)
- [Brain-like representational straightening of natural movies in robust feedforward neural networks](https://mlsanity.com/paper/2308.13870/) (2023)
- [Selective Volume Mixup for Video Action Recognition](https://mlsanity.com/paper/2309.09534/) (2023)

## Abstract (arXiv)

In recent years, a number of approaches based on 2D or 3D convolutional neural networks (CNN) have emerged for video action recognition, achieving state-of-the-art results on several large-scale benchmark datasets. In this paper, we carry out in-depth comparative analysis to better understand the differences between these approaches and the progress made by them. To this end, we develop an unified framework for both 2D-CNN and 3D-CNN action models, which enables us to remove bells and whistles and provides a common ground for fair comparison. We then conduct an effort towards a large-scale analysis involving over 300 action recognition models. Our comprehensive analysis reveals that a) a significant leap is made in efficiency for action recognition, but not in accuracy; b) 2D-CNN and 3D-CNN models behave similarly in terms of spatio-temporal representation abilities and transferability. Our codes are available at https://github.com/IBM/action-recognition-pytorch.