# Video Classification with Channel-Separated Convolutional Networks

arXiv: https://arxiv.org/abs/1904.02811 · HTML: https://mlsanity.com/paper/1904.02811/

## Key facts

- Cited by 290 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2019.

## Built on by (most cited first, 30 of 290)

- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Revisiting Skeleton-based Action Recognition](https://mlsanity.com/paper/2104.13586/) (2021)
- [Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers](https://mlsanity.com/paper/2106.05392/) (2021)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://mlsanity.com/paper/2201.09450/) (2022)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2206.13559/) (2022)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning](https://mlsanity.com/paper/2201.04676/) (2022)
- [Adaptive Token Sampling For Efficient Vision Transformers](https://mlsanity.com/paper/2111.15667/) (2021)
- [Frozen CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2208.03550/) (2022)
- [BEVT: BERT Pretraining of Video Transformers](https://mlsanity.com/paper/2112.01529/) (2021)
- [Multiview Transformers for Video Recognition](https://mlsanity.com/paper/2201.04288/) (2022)
- [Towards Long-Form Video Understanding](https://mlsanity.com/paper/2106.11310/) (2021)
- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation](https://mlsanity.com/paper/2104.04691/) (2021)
- [Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection](https://mlsanity.com/paper/2201.07131/) (2022)
- [HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling](https://mlsanity.com/paper/2204.13686/) (2022)
- [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](https://mlsanity.com/paper/2211.09552/) (2022)
- [VideoLLM: Modeling Video Sequence with Large Language Models](https://mlsanity.com/paper/2305.13292/) (2023)
- [VidTr: Video Transformer Without Convolutions](https://mlsanity.com/paper/2104.11746/) (2021)
- [Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding](https://mlsanity.com/paper/2403.09626/) (2024)
- [Long Movie Clip Classification with State-Space Video Models](https://mlsanity.com/paper/2204.01692/) (2022)

## Abstract (arXiv)

Group convolution has been shown to offer great computational savings in various 2D convolutional architectures for image classification. It is natural to ask: 1) if group convolution can help to alleviate the high computational cost of video classification networks; 2) what factors matter the most in 3D group convolutional networks; and 3) what are good computation/accuracy trade-offs with 3D group convolutional networks. This paper studies the effects of different design choices in 3D group convolutional networks for video classification. We empirically demonstrate that the amount of channel interactions plays an important role in the accuracy of 3D group convolutional networks. Our experiments suggest two main findings. First, it is a good practice to factorize 3D convolutions by separating channel interactions and spatiotemporal interactions as this leads to improved accuracy and lower computational cost. Second, 3D channel-separated convolutions provide a form of regularization, yielding lower training accuracy but higher test accuracy compared to 3D convolutions. These two empirical findings lead us to design an architecture -- Channel-Separated Convolutional Network (CSN) -- which is simple, efficient, yet accurate. On Sports1M, Kinetics, and Something-Something, our CSNs are comparable with or better than the state-of-the-art while being 2-3 times more efficient.