# CT-Net: Channel Tensorization Network for Video Classification

arXiv: https://arxiv.org/abs/2106.01603 · HTML: https://mlsanity.com/paper/2106.01603/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Videos as Space-Time Region Graphs](https://mlsanity.com/paper/1806.01810/) (2018)
- [TEA: Temporal Excitation and Aggregation for Action Recognition](https://mlsanity.com/paper/2004.01398/) (2020)
- [STM: SpatioTemporal and Motion Encoding for Action Recognition](https://mlsanity.com/paper/1908.02486/) (2019)
- [ECO: Efficient Convolutional Network for Online Video Understanding](https://mlsanity.com/paper/1804.09066/) (2018)
- [TEINet: Towards an Efficient Architecture for Video Recognition](https://mlsanity.com/paper/1911.09435/) (2019)
- [Gate-Shift Networks for Video Action Recognition](https://mlsanity.com/paper/1912.00381/) (2019)
- [Video Modeling with Correlation Networks](https://mlsanity.com/paper/1906.03349/) (2019)
- [SmallBigNet: Integrating Core and Contextual Views for Video Classification](https://mlsanity.com/paper/2006.14582/) (2020)
- [Saliency Tubes: Visual Explanations for Spatio-Temporal Convolutions](https://mlsanity.com/paper/1902.01078/) (2019)
- [Temporal Distinct Representation Learning for Action Recognition](https://mlsanity.com/paper/2007.07626/) (2020)

## Built on by (most cited first, 26 of 26)

- [VideoMamba: State Space Model for Efficient Video Understanding](https://mlsanity.com/paper/2403.06977/) (2024)
- [UniFormer: Unifying Convolution and Self-attention for Visual Recognition](https://mlsanity.com/paper/2201.09450/) (2022)
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2206.13559/) (2022)
- [UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning](https://mlsanity.com/paper/2201.04676/) (2022)
- [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](https://mlsanity.com/paper/2211.09552/) (2022)
- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition](https://mlsanity.com/paper/2107.08580/) (2021)
- [ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting](https://mlsanity.com/paper/2209.00065/) (2022)
- [MorphMLP: An Efficient MLP-Like Backbone for Spatial-Temporal Representation Learning](https://mlsanity.com/paper/2111.12527/) (2021)
- [Diversifying Spatial-Temporal Perception for Video Domain Generalization](https://mlsanity.com/paper/2310.17942/) (2023)
- [Relational Self-Attention: What's Missing in Attention for Video Understanding](https://mlsanity.com/paper/2111.01673/) (2021)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [LAC: Latent Action Composition for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2308.14500/) (2023)
- [XOV-Action: Towards Generalizable Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2403.01560/) (2024)
- [Learning Correlation Structures for Vision Transformers](https://mlsanity.com/paper/2404.03924/) (2024)
- [Human-Centric Transformer for Domain Adaptive Action Recognition](https://mlsanity.com/paper/2407.10860/) (2024)
- [Spatial-temporal Transformer-guided Diffusion based Data Augmentation for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.13434/) (2023)
- [Transformed ROIs for Capturing Visual Transformations in Videos](https://mlsanity.com/paper/2106.03162/) (2021)
- [PosMLP-Video: Spatial and Temporal Relative Position Encoding for Efficient Video Recognition](https://mlsanity.com/paper/2407.02934/) (2024)
- [Efficient Attention-free Video Shift Transformers](https://mlsanity.com/paper/2208.11108/) (2022)
- [In Defense of Image Pre-Training for Spatiotemporal Recognition](https://mlsanity.com/paper/2205.01721/) (2022)
- [An end-to-end multi-scale network for action prediction in videos](https://mlsanity.com/paper/2301.01216/) (2022)
- [ConFormer: A Novel Collection of Deep Learning Models to Assist Cardiologists in the Assessment of Cardiac Function](https://mlsanity.com/paper/2312.08567/) (2023)
- [Hierarchical Action Recognition: A Contrastive Video-Language Approach with Hierarchical Interactions](https://mlsanity.com/paper/2405.17729/) (2024)
- [StableMamba: Distillation-free Scaling of Large SSMs for Images and Videos](https://mlsanity.com/paper/2409.11867/) (2024)
- [VideoSEMA: a scalable and efficient Mamba-like attention for video understanding](https://mlsanity.com/paper/2607.14711/) (2026)

## Abstract (arXiv)

3D convolution is powerful for video classification but often computationally expensive, recent studies mainly focus on decomposing it on spatial-temporal and/or channel dimensions. Unfortunately, most approaches fail to achieve a preferable balance between convolutional efficiency and feature-interaction sufficiency. For this reason, we propose a concise and novel Channel Tensorization Network (CT-Net), by treating the channel dimension of input feature as a multiplication of K sub-dimensions. On one hand, it naturally factorizes convolution in a multiple dimension way, leading to a light computation burden. On the other hand, it can effectively enhance feature interaction from different channels, and progressively enlarge the 3D receptive field of such interaction to boost classification accuracy. Furthermore, we equip our CT-Module with a Tensor Excitation (TE) mechanism. It can learn to exploit spatial, temporal and channel attention in a high-dimensional manner, to improve the cooperative power of all the feature dimensions in our CT-Module. Finally, we flexibly adapt ResNet as our CT-Net. Extensive experiments are conducted on several challenging video benchmarks, e.g., Kinetics-400, Something-Something V1 and V2. Our CT-Net outperforms a number of recent SOTA approaches, in terms of accuracy and/or efficiency. The codes and models will be available on https://github.com/Andy1621/CT-Net.