# F4D: Factorized 4D Convolutional Neural Network for Efficient Video-level Representation Learning

arXiv: https://arxiv.org/abs/2401.08609 · HTML: https://mlsanity.com/paper/2401.08609/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Temporal Relational Reasoning in Videos](https://mlsanity.com/paper/1711.08496/) (2017)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [TEA: Temporal Excitation and Aggregation for Action Recognition](https://mlsanity.com/paper/2004.01398/) (2020)
- [STM: SpatioTemporal and Motion Encoding for Action Recognition](https://mlsanity.com/paper/1908.02486/) (2019)
- [ECO: Efficient Convolutional Network for Online Video Understanding](https://mlsanity.com/paper/1804.09066/) (2018)
- [Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition](https://mlsanity.com/paper/1711.11152/) (2017)
- [MotionSqueeze: Neural Motion Feature Learning for Video Understanding](https://mlsanity.com/paper/2007.09933/) (2020)
- [Appearance-and-Relation Networks for Video Classification](https://mlsanity.com/paper/1711.09125/) (2017)
- [Real-time Action Recognition with Enhanced Motion Vector CNNs](https://mlsanity.com/paper/1604.07669/) (2016)
- [Representation Flow for Action Recognition](https://mlsanity.com/paper/1810.01455/) (2018)
- [ConvNet Architecture Search for Spatiotemporal Feature Learning](https://mlsanity.com/paper/1708.05038/) (2017)
- [Spatio-Temporal Channel Correlation Networks for Action Classification](https://mlsanity.com/paper/1806.07754/) (2018)

## Built on by (most cited first, 2 of 2)

- [Enhancing Environmental Robustness in Few-shot Learning via Conditional Representation Learning](https://mlsanity.com/paper/2502.01183/) (2025)
- [Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenization](https://mlsanity.com/paper/2509.05695/) (2025)

## Abstract (arXiv)

Recent studies have shown that video-level representation learning is crucial to the capture and understanding of the long-range temporal structure for video action recognition. Most existing 3D convolutional neural network (CNN)-based methods for video-level representation learning are clip-based and focus only on short-term motion and appearances. These CNN-based methods lack the capacity to incorporate and model the long-range spatiotemporal representation of the underlying video and ignore the long-range video-level context during training. In this study, we propose a factorized 4D CNN architecture with attention (F4D) that is capable of learning more effective, finer-grained, long-term spatiotemporal video representations. We demonstrate that the proposed F4D architecture yields significant performance improvements over the conventional 2D, and 3D CNN architectures proposed in the literature. Experiment evaluation on five action recognition benchmark datasets, i.e., Something-Something-v1, SomethingSomething-v2, Kinetics-400, UCF101, and HMDB51 demonstrate the effectiveness of the proposed F4D network architecture for video-level action recognition.