# Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

arXiv: https://arxiv.org/abs/1504.01561 · HTML: https://mlsanity.com/paper/1504.01561/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2015.

## Built on by (most cited first, 21 of 21)

- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [OmniVid: A Generative Framework for Universal Video Understanding](https://mlsanity.com/paper/2403.17935/) (2024)
- [Efficient Robustness Assessment via Adversarial Spatial-Temporal Focus on Videos](https://mlsanity.com/paper/2301.00896/) (2023)
- [Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI](https://mlsanity.com/paper/2312.14229/) (2023)
- [SCAN: A Spatial Context Attentive Network for Joint Multi-Agent Intent Prediction](https://mlsanity.com/paper/2102.00109/) (2021)
- [PGT: A Progressive Method for Training Models on Long Videos](https://mlsanity.com/paper/2103.11313/) (2021)
- [Supervised Contrastive Learning for Detecting Anomalous Driving Behaviours from Multimodal Videos](https://mlsanity.com/paper/2109.04021/) (2021)
- [CyclingNet: Detecting cycling near misses from video streams in complex urban scenes with deep learning](https://mlsanity.com/paper/2102.00565/) (2021)
- [Driving Style Representation in Convolutional Recurrent Neural Network Model of Driver Identification](https://mlsanity.com/paper/2102.05843/) (2021)
- [Spatio-Temporal Representation Learning Enhanced Source Cell-phone Recognition from Speech Recordings](https://mlsanity.com/paper/2208.12753/) (2022)
- [TempT: Temporal consistency for Test-time adaptation](https://mlsanity.com/paper/2303.10536/) (2023)
- [RNNs, CNNs and Transformers in Human Action Recognition: A Survey and a Hybrid Model](https://mlsanity.com/paper/2407.06162/) (2024)
- [Two-Stream Transformer Architecture for Long Video Understanding](https://mlsanity.com/paper/2208.01753/) (2022)
- [SkateboardAI: The Coolest Video Action Recognition for Skateboarding](https://mlsanity.com/paper/2311.11467/) (2023)
- [Co-segmentation Inspired Attention Module for Video-based Computer Vision Tasks](https://mlsanity.com/paper/2111.07370/) (2021)
- [Rabbit, toad, and the Moon: Can machine categorize them into one class?](https://mlsanity.com/paper/2203.16163/) (2022)
- [Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework](https://mlsanity.com/paper/2208.05034/) (2022)
- [Transferable-guided Attention Is All You Need for Video Domain Adaptation](https://mlsanity.com/paper/2407.01375/) (2024)
- [Exploring Automated Recognition of Instructional Activity and Discourse from Multimodal Classroom Data](https://mlsanity.com/paper/2512.00087/) (2025)
- [CLTA: Contents and Length-based Temporal Attention for Few-shot Action Recognition](https://mlsanity.com/paper/2103.10567/) (2021)
- [Video Imprint](https://mlsanity.com/paper/2106.03283/) (2021)

## Abstract (arXiv)

Classifying videos according to content semantics is an important problem with a wide range of applications. In this paper, we propose a hybrid deep learning framework for video classification, which is able to model static spatial information, short-term motion, as well as long-term temporal clues in the videos. Specifically, the spatial and the short-term motion features are extracted separately by two Convolutional Neural Networks (CNN). These two types of CNN-based features are then combined in a regularized feature fusion network for classification, which is able to learn and utilize feature relationships for improved performance. In addition, Long Short Term Memory (LSTM) networks are applied on top of the two features to further model longer-term temporal clues. The main contribution of this work is the hybrid learning framework that can model several important aspects of the video data. We also show that (1) combining the spatial and the short-term motion features in the regularized fusion network is better than direct classification and fusion using the CNN with a softmax layer, and (2) the sequence-based LSTM is highly complementary to the traditional classification strategy without considering the temporal frame orders. Extensive experiments are conducted on two popular and challenging benchmarks, the UCF-101 Human Actions and the Columbia Consumer Videos (CCV). On both benchmarks, our framework achieves to-date the best reported performance: 91.3% on the UCF-101 and 83.5% on the CCV.