# Gate-Shift Networks for Video Action Recognition

arXiv: https://arxiv.org/abs/1912.00381 · HTML: https://mlsanity.com/paper/1912.00381/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2019.

## Built on by (most cited first, 30 of 76)

- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [ACTION-Net: Multipath Excitation for Action Recognition](https://mlsanity.com/paper/2103.07372/) (2021)
- [MGSampler: An Explainable Sampling Strategy for Video Action Recognition](https://mlsanity.com/paper/2104.09952/) (2021)
- [AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition](https://mlsanity.com/paper/2102.05775/) (2021)
- [ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detection](https://mlsanity.com/paper/2310.20208/) (2023)
- [SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training](https://mlsanity.com/paper/2307.08476/) (2023)
- [Spotting Temporally Precise, Fine-Grained Events in Video](https://mlsanity.com/paper/2207.10213/) (2022)
- [AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition](https://mlsanity.com/paper/2112.14238/) (2021)
- [Dynamic Network Quantization for Efficient Video Inference](https://mlsanity.com/paper/2108.10394/) (2021)
- [Domain Generalization through Audio-Visual Relative Norm Alignment in First Person Action Recognition](https://mlsanity.com/paper/2110.10101/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition](https://mlsanity.com/paper/2102.07092/) (2021)
- [Video Pose Distillation for Few-Shot, Fine-Grained Sports Action Recognition](https://mlsanity.com/paper/2109.01305/) (2021)
- [CT-Net: Channel Tensorization Network for Video Classification](https://mlsanity.com/paper/2106.01603/) (2021)
- [No frame left behind: Full Video Action Recognition](https://mlsanity.com/paper/2103.15395/) (2021)
- [T-DEED: Temporal-Discriminability Enhancer Encoder-Decoder for Precise Event Spotting in Sports Videos](https://mlsanity.com/paper/2404.05392/) (2024)
- [Stand-Alone Inter-Frame Attention in Video Models](https://mlsanity.com/paper/2206.06931/) (2022)
- [Diversifying Spatial-Temporal Perception for Video Domain Generalization](https://mlsanity.com/paper/2310.17942/) (2023)
- [Revisiting 3D Object Detection From an Egocentric Perspective](https://mlsanity.com/paper/2112.07787/) (2021)
- [Group Contextualization for Video Recognition](https://mlsanity.com/paper/2203.09694/) (2022)
- [AdaFocusV3: On Unified Spatial-temporal Dynamic Video Recognition](https://mlsanity.com/paper/2209.13465/) (2022)
- [Dynamic Spatio-Temporal Specialization Learning for Fine-Grained Action Recognition](https://mlsanity.com/paper/2209.01425/) (2022)
- [ASTRA: An Action Spotting TRAnsformer for Soccer Videos](https://mlsanity.com/paper/2404.01891/) (2024)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [Backbones-Review: Feature Extraction Networks for Deep Learning and Deep Reinforcement Learning Approaches](https://mlsanity.com/paper/2206.08016/) (2022)
- [XOV-Action: Towards Generalizable Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2403.01560/) (2024)
- [Towards Active Learning for Action Spotting in Association Football Videos](https://mlsanity.com/paper/2304.04220/) (2023)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [DSANet: Dynamic Segment Aggregation Network for Video-Level Representation Learning](https://mlsanity.com/paper/2105.12085/) (2021)
- [Attention in Attention: Modeling Context Correlation for Efficient Video Classification](https://mlsanity.com/paper/2204.09303/) (2022)

## Abstract (arXiv)

Deep 3D CNNs for video action recognition are designed to learn powerful representations in the joint spatio-temporal feature space. In practice however, because of the large number of parameters and computations involved, they may under-perform in the lack of sufficiently large datasets for training them at scale. In this paper we introduce spatial gating in spatial-temporal decomposition of 3D kernels. We implement this concept with Gate-Shift Module (GSM). GSM is lightweight and turns a 2D-CNN into a highly efficient spatio-temporal feature extractor. With GSM plugged in, a 2D-CNN learns to adaptively route features through time and combine them, at almost no additional parameters and computational overhead. We perform an extensive evaluation of the proposed module to study its effectiveness in video action recognition, achieving state-of-the-art results on Something Something-V1 and Diving48 datasets, and obtaining competitive results on EPIC-Kitchens with far less model complexity.