# Collaborative Spatio-temporal Feature Learning for Video Action Recognition

arXiv: https://arxiv.org/abs/1903.01197 · HTML: https://mlsanity.com/paper/1903.01197/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2019.

## Built on by (most cited first, 24 of 24)

- [Holistic Interaction Transformer Network for Action Detection](https://mlsanity.com/paper/2210.12686/) (2022)
- [Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition](https://mlsanity.com/paper/2112.10992/) (2021)
- [Video Summarization through Reinforcement Learning with a 3D Spatio-Temporal U-Net](https://mlsanity.com/paper/2106.10528/) (2021)
- [Can An Image Classifier Suffice For Action Recognition?](https://mlsanity.com/paper/2106.14104/) (2021)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [Spatiotemporal Self-attention Modeling with Temporal Patch Shift for Action Recognition](https://mlsanity.com/paper/2207.13259/) (2022)
- [GTNet:Guided Transformer Network for Detecting Human-Object Interactions](https://mlsanity.com/paper/2108.00596/) (2021)
- [Hand-Object Interaction Reasoning](https://mlsanity.com/paper/2201.04906/) (2022)
- [Interaction-Aware Prompting for Zero-Shot Spatio-Temporal Action Detection](https://mlsanity.com/paper/2304.04688/) (2023)
- [MAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformers](https://mlsanity.com/paper/2308.03741/) (2023)
- [MAiVAR: Multimodal Audio-Image and Video Action Recognizer](https://mlsanity.com/paper/2209.04780/) (2022)
- [A Closer Look at Spatial-Slice Features Learning for COVID-19 Detection](https://mlsanity.com/paper/2404.01643/) (2024)
- [REGINA - Reasoning Graph Convolutional Networks in Human Action Recognition](https://mlsanity.com/paper/2105.06711/) (2021)
- [A Dataset and Model for Crossing Indian Roads](https://mlsanity.com/paper/2211.07916/) (2022)
- [Efficient Spatialtemporal Context Modeling for Action Recognition](https://mlsanity.com/paper/2103.11190/) (2021)
- [NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition](https://mlsanity.com/paper/2104.01110/) (2021)
- [MAVR-Net: Robust Multi-View Learning for MAV Action Recognition with Cross-View Attention](https://mlsanity.com/paper/2510.15448/) (2025)
- [SMV-EAR: Bring Spatiotemporal Multi-View Representation Learning into Efficient Event-Based Action Recognition](https://mlsanity.com/paper/2601.17391/) (2026)
- [HoloEv-Net: Efficient Event-based Action Recognition via Holographic Spatial Embedding and Global Spectral Gating](https://mlsanity.com/paper/2602.04182/) (2026)
- [Position: Life-Logging Video Streams Make the Privacy-Utility Trade-off Inevitable](https://mlsanity.com/paper/2605.10404/) (2026)
- [Spatiotemporal Modeling Encounters 3D Medical Image Analysis: Slice-Shift UNet with Multi-View Fusion](https://mlsanity.com/paper/2307.12853/) (2023)
- [MobiAct: Efficient MAV Action Recognition Using MobileNetV4 with Contrastive Learning and Knowledge Distillation](https://mlsanity.com/paper/2510.19273/) (2025)
- [Novel Semantic Prompting for Zero-Shot Action Recognition](https://mlsanity.com/paper/2603.08289/) (2026)
- [Insights from Visual Cognition: Understanding Human Action Dynamics with Overall Glance and Refined Gaze Transformer](https://mlsanity.com/paper/2604.06783/) (2026)

## Abstract (arXiv)

Spatio-temporal feature learning is of central importance for action recognition in videos. Existing deep neural network models either learn spatial and temporal features independently (C2D) or jointly with unconstrained parameters (C3D). In this paper, we propose a novel neural operation which encodes spatio-temporal features collaboratively by imposing a weight-sharing constraint on the learnable parameters. In particular, we perform 2D convolution along three orthogonal views of volumetric video data,which learns spatial appearance and temporal motion cues respectively. By sharing the convolution kernels of different views, spatial and temporal features are collaboratively learned and thus benefit from each other. The complementary features are subsequently fused by a weighted summation whose coefficients are learned end-to-end. Our approach achieves state-of-the-art performance on large-scale benchmarks and won the 1st place in the Moments in Time Challenge 2018. Moreover, based on the learned coefficients of different views, we are able to quantify the contributions of spatial and temporal features. This analysis sheds light on interpretability of the model and may also guide the future design of algorithm for video recognition.