# ConvNet Architecture Search for Spatiotemporal Feature Learning

arXiv: https://arxiv.org/abs/1708.05038 · HTML: https://mlsanity.com/paper/1708.05038/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2017.

## Built on by (most cited first, 30 of 40)

- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding](https://mlsanity.com/paper/2403.09626/) (2024)
- [FrameExit: Conditional Early Exiting for Efficient Video Recognition](https://mlsanity.com/paper/2104.13400/) (2021)
- [MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition](https://mlsanity.com/paper/2108.09322/) (2021)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [DCAN: Improving Temporal Action Detection via Dual Context Aggregation](https://mlsanity.com/paper/2112.03612/) (2021)
- [Aligning Correlation Information for Domain Adaptation in Action Recognition](https://mlsanity.com/paper/2107.04932/) (2021)
- [Coarse-Fine Networks for Temporal Activity Detection in Videos](https://mlsanity.com/paper/2103.01302/) (2021)
- [BasicTAD: an Astounding RGB-Only Baseline for Temporal Action Detection](https://mlsanity.com/paper/2205.02717/) (2022)
- [Learning from Temporal Gradient for Semi-supervised Action Recognition](https://mlsanity.com/paper/2111.13241/) (2021)
- [Expansion-Squeeze-Excitation Fusion Network for Elderly Activity Recognition](https://mlsanity.com/paper/2112.10992/) (2021)
- [Bullying10K: A Large-Scale Neuromorphic Dataset towards Privacy-Preserving Bullying Recognition](https://mlsanity.com/paper/2306.11546/) (2023)
- [SVIP: Sequence VerIfication for Procedures in Videos](https://mlsanity.com/paper/2112.06447/) (2021)
- [End-to-End Compressed Video Representation Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2203.15336/) (2022)
- [GL-RG: Global-Local Representation Granularity for Video Captioning](https://mlsanity.com/paper/2205.10706/) (2022)
- [Deep Learning for Video-Text Retrieval: a Review](https://mlsanity.com/paper/2302.12552/) (2023)
- [Deep Unsupervised Key Frame Extraction for Efficient Video Classification](https://mlsanity.com/paper/2211.06742/) (2022)
- [Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2408.09764/) (2024)
- [Hierarchical Compositional Representations for Few-shot Action Recognition](https://mlsanity.com/paper/2208.09424/) (2022)
- [EZSR: Event-based Zero-Shot Recognition](https://mlsanity.com/paper/2407.21616/) (2024)
- [A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities](https://mlsanity.com/paper/2408.05436/) (2024)
- [Lane Change Classification and Prediction with Action Recognition Networks](https://mlsanity.com/paper/2208.11650/) (2022)
- [Local Compressed Video Stream Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2309.15431/) (2023)
- [Retain, Blend, and Exchange: A Quality-aware Spatial-Stereo Fusion Approach for Event Stream Recognition](https://mlsanity.com/paper/2406.18845/) (2024)
- [ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition](https://mlsanity.com/paper/2409.01564/) (2024)
- [DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection](https://mlsanity.com/paper/2510.22622/) (2025)
- [F4D: Factorized 4D Convolutional Neural Network for Efficient Video-level Representation Learning](https://mlsanity.com/paper/2401.08609/) (2023)
- [Efficient Spatialtemporal Context Modeling for Action Recognition](https://mlsanity.com/paper/2103.11190/) (2021)
- [Efficient Human Vision Inspired Action Recognition using Adaptive Spatiotemporal Sampling](https://mlsanity.com/paper/2207.05249/) (2022)
- [Adversarial Domain Adaptation for Action Recognition Around the Clock](https://mlsanity.com/paper/2210.17412/) (2022)

## Abstract (arXiv)

Learning image representations with ConvNets by pre-training on ImageNet has proven useful across many visual understanding tasks including object detection, semantic segmentation, and image captioning. Although any image representation can be applied to video frames, a dedicated spatiotemporal representation is still vital in order to incorporate motion patterns that cannot be captured by appearance based models alone. This paper presents an empirical ConvNet architecture search for spatiotemporal feature learning, culminating in a deep 3-dimensional (3D) Residual ConvNet. Our proposed architecture outperforms C3D by a good margin on Sports-1M, UCF101, HMDB51, THUMOS14, and ASLAN while being 2 times faster at inference time, 2 times smaller in model size, and having a more compact representation.