# Evolving Space-Time Neural Architectures for Videos

arXiv: https://arxiv.org/abs/1811.10636 · HTML: https://mlsanity.com/paper/1811.10636/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2018.

## Built on by (most cited first, 17 of 17)

- [TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?](https://mlsanity.com/paper/2106.11297/) (2021)
- [Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning](https://mlsanity.com/paper/2212.03229/) (2022)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [VicTR: Video-conditioned Text Representations for Activity Recognition](https://mlsanity.com/paper/2304.02560/) (2023)
- [ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search](https://mlsanity.com/paper/2105.10154/) (2021)
- [Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition](https://mlsanity.com/paper/2101.06636/) (2021)
- [MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurations](https://mlsanity.com/paper/2105.07085/) (2021)
- [Medical Imaging and Machine Learning](https://mlsanity.com/paper/2103.01938/) (2021)
- [Flow Dynamics Correction for Action Recognition](https://mlsanity.com/paper/2310.10059/) (2023)
- [AutoVideo: An Automated Video Action Recognition System](https://mlsanity.com/paper/2108.04212/) (2021)
- [Self-Supervised Learning via multi-Transformation Classification for Action Recognition](https://mlsanity.com/paper/2102.10378/) (2021)
- [Visionary: Vision architecture discovery for robot learning](https://mlsanity.com/paper/2103.14633/) (2021)
- [Adaptive Intermediate Representations for Video Understanding](https://mlsanity.com/paper/2104.07135/) (2021)
- [Feature Hallucination for Self-supervised Action Recognition](https://mlsanity.com/paper/2506.20342/) (2025)
- [NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition](https://mlsanity.com/paper/2104.01110/) (2021)
- [Meta-Learning of NAS for Few-shot Learning in Medical Image Applications](https://mlsanity.com/paper/2203.08951/) (2022)
- [CycleGANAS: Differentiable Neural Architecture Search for CycleGAN](https://mlsanity.com/paper/2311.07162/) (2023)

## Abstract (arXiv)

We present a new method for finding video CNN architectures that capture rich spatio-temporal information in videos. Previous work, taking advantage of 3D convolutions, obtained promising results by manually designing video CNN architectures. We here develop a novel evolutionary search algorithm that automatically explores models with different types and combinations of layers to jointly learn interactions between spatial and temporal aspects of video representations. We demonstrate the generality of this algorithm by applying it to two meta-architectures, obtaining new architectures superior to manually designed architectures. Further, we propose a new component, the iTGM layer, which more efficiently utilizes its parameters to allow learning of space-time interactions over longer time horizons. The iTGM layer is often preferred by the evolutionary algorithm and allows building cost-efficient networks. The proposed approach discovers new and diverse video architectures that were previously unknown. More importantly they are both more accurate and faster than prior models, and outperform the state-of-the-art results on multiple datasets we test, including HMDB, Kinetics, and Moments in Time. We will open source the code and models, to encourage future model development.