# Learning Efficient Video Representation with Video Shuffle Networks

arXiv: https://arxiv.org/abs/1911.11319 · HTML: https://mlsanity.com/paper/1911.11319/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2019.

## Built on by (most cited first, 3 of 3)

- [Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion](https://mlsanity.com/paper/2108.05009/) (2021)
- [Auto-X3D: Ultra-Efficient Video Understanding via Finer-Grained Neural Architecture Search](https://mlsanity.com/paper/2112.04710/) (2021)
- [An Efficient 3D Convolutional Neural Network with Channel-wise, Spatial-grouped, and Temporal Convolutions](https://mlsanity.com/paper/2503.00796/) (2025)

## Abstract (arXiv)

3D CNN shows its strong ability in learning spatiotemporal representation in recent video recognition tasks. However, inflating 2D convolution to 3D inevitably introduces additional computational costs, making it cumbersome in practical deployment. We consider whether there is a way to equip the conventional 2D convolution with temporal vision no requiring expanding its kernel. To this end, we propose the video shuffle, a parameter-free plug-in component that efficiently reallocates the inputs of 2D convolution so that its receptive field can be extended to the temporal dimension. In practical, video shuffle firstly divides each frame feature into multiple groups and then aggregate the grouped features via temporal shuffle operation. This allows the following 2D convolution aggregate the global spatiotemporal features. The proposed video shuffle can be flexibly inserted into popular 2D CNNs, forming the Video Shuffle Networks (VSN). With a simple yet efficient implementation, VSN performs surprisingly well on temporal modeling benchmarks. In experiments, VSN not only gains non-trivial improvements on Kinetics and Moments in Time, but also achieves state-of-the-art performance on Something-Something-V1, Something-Something-V2 datasets.