# Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks

arXiv: https://arxiv.org/abs/1510.00562 · HTML: https://mlsanity.com/paper/1510.00562/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2015.

## Built on by (most cited first, 30 of 40)

- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Learning to Act from Actionless Videos through Dense Correspondences](https://mlsanity.com/paper/2310.08576/) (2023)
- [Multiview Transformers for Video Recognition](https://mlsanity.com/paper/2201.04288/) (2022)
- [E2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition](https://mlsanity.com/paper/2112.03596/) (2021)
- [HAKE: A Knowledge Engine Foundation for Human Activity Understanding](https://mlsanity.com/paper/2202.06851/) (2022)
- [Holistic Interaction Transformer Network for Action Detection](https://mlsanity.com/paper/2210.12686/) (2022)
- [Augmented Transformer with Adaptive Graph for Temporal Action Proposal Generation](https://mlsanity.com/paper/2103.16024/) (2021)
- [ACSNet: Action-Context Separation Network for Weakly Supervised Temporal Action Localization](https://mlsanity.com/paper/2103.15088/) (2021)
- [Skeleton-Based Action Segmentation with Multi-Stage Spatial-Temporal Graph Convolutional Neural Networks](https://mlsanity.com/paper/2202.01727/) (2022)
- [Hypergraph-based Multi-View Action Recognition using Event Cameras](https://mlsanity.com/paper/2403.19316/) (2024)
- [Video-FocalNets: Spatio-Temporal Focal Modulation for Video Action Recognition](https://mlsanity.com/paper/2307.06947/) (2023)
- [TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Device](https://mlsanity.com/paper/2109.13227/) (2021)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [Deep Learning for Video-Text Retrieval: a Review](https://mlsanity.com/paper/2302.12552/) (2023)
- [Deep Unsupervised Key Frame Extraction for Efficient Video Classification](https://mlsanity.com/paper/2211.06742/) (2022)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories](https://mlsanity.com/paper/2104.01198/) (2021)
- [Enhancing Temporal Modeling of Video LLMs via Time Gating](https://mlsanity.com/paper/2410.05714/) (2024)
- [A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future](https://mlsanity.com/paper/2412.14056/) (2024)
- [PGT: A Progressive Method for Training Models on Long Videos](https://mlsanity.com/paper/2103.11313/) (2021)
- [Video-based Human Action Recognition using Deep Learning: A Review](https://mlsanity.com/paper/2208.03775/) (2022)
- [A transformer-based model for default prediction in mid-cap corporate markets](https://mlsanity.com/paper/2111.09902/) (2021)
- [ADNet: Temporal Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/2104.06653/) (2021)
- [Joint Learning On The Hierarchy Representation for Fine-Grained Human Action Recognition](https://mlsanity.com/paper/2110.05853/) (2021)
- [Joint Temporal Pooling for Improving Skeleton-based Action Recognition](https://mlsanity.com/paper/2408.09356/) (2024)
- [Extending Temporal Data Augmentation for Video Action Recognition](https://mlsanity.com/paper/2211.04888/) (2022)
- [SSTM: Spatiotemporal Recurrent Transformers for Multi-frame Optical Flow Estimation](https://mlsanity.com/paper/2304.14418/) (2023)
- [Towards Scalable Modeling of Compressed Videos for Efficient Action Recognition](https://mlsanity.com/paper/2503.13724/) (2025)
- [Early Melanoma Diagnosis with Sequential Dermoscopic Images](https://mlsanity.com/paper/2110.05976/) (2021)
- [DOAD: Decoupled One Stage Action Detection Network](https://mlsanity.com/paper/2304.00254/) (2023)

## Abstract (arXiv)

Human actions in video sequences are three-dimensional (3D) spatio-temporal signals characterizing both the visual appearance and motion dynamics of the involved humans and objects. Inspired by the success of convolutional neural networks (CNN) for image classification, recent attempts have been made to learn 3D CNNs for recognizing human actions in videos. However, partly due to the high complexity of training 3D convolution kernels and the need for large quantities of training videos, only limited success has been reported. This has triggered us to investigate in this paper a new deep architecture which can handle 3D signals more effectively. Specifically, we propose factorized spatio-temporal convolutional networks (FstCN) that factorize the original 3D convolution kernel learning as a sequential process of learning 2D spatial kernels in the lower layers (called spatial convolutional layers), followed by learning 1D temporal kernels in the upper layers (called temporal convolutional layers). We introduce a novel transformation and permutation operator to make factorization in FstCN possible. Moreover, to address the issue of sequence alignment, we propose an effective training and inference strategy based on sampling multiple video clips from a given action video sequence. We have tested FstCN on two commonly used benchmark datasets (UCF-101 and HMDB-51). Without using auxiliary training videos to boost the performance, FstCN outperforms existing CNN based methods and achieves comparable performance with a recent method that benefits from using auxiliary training videos.