# V4D:4D Convolutional Neural Networks for Video-level Representation Learning

arXiv: https://arxiv.org/abs/2002.07442 · HTML: https://mlsanity.com/paper/2002.07442/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2020.

## Built on by (most cited first, 30 of 33)

- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [MGSampler: An Explainable Sampling Strategy for Video Action Recognition](https://mlsanity.com/paper/2104.09952/) (2021)
- [Video Contrastive Learning with Global Context](https://mlsanity.com/paper/2108.02722/) (2021)
- [DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection](https://mlsanity.com/paper/2306.00863/) (2023)
- [No Pain, Big Gain: Classify Dynamic Point Cloud Sequences with Static Models by Fitting Feature-level Space-time Surfaces](https://mlsanity.com/paper/2203.11113/) (2022)
- [Diversifying Spatial-Temporal Perception for Video Domain Generalization](https://mlsanity.com/paper/2310.17942/) (2023)
- [Group Contextualization for Video Recognition](https://mlsanity.com/paper/2203.09694/) (2022)
- [Unsupervised Motion Representation Learning with Capsule Autoencoders](https://mlsanity.com/paper/2110.00529/) (2021)
- [Gate-Shift-Fuse for Video Action Recognition](https://mlsanity.com/paper/2203.08897/) (2022)
- [XOV-Action: Towards Generalizable Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2403.01560/) (2024)
- [DSANet: Dynamic Segment Aggregation Network for Video-Level Representation Learning](https://mlsanity.com/paper/2105.12085/) (2021)
- [TFCNet: Temporal Fully Connected Networks for Static Unbiased Temporal Reasoning](https://mlsanity.com/paper/2203.05928/) (2022)
- [Attention in Attention: Modeling Context Correlation for Efficient Video Classification](https://mlsanity.com/paper/2204.09303/) (2022)
- [What Can Simple Arithmetic Operations Do for Temporal Modeling?](https://mlsanity.com/paper/2307.08908/) (2023)
- [Temporal-attentive Covariance Pooling Networks for Video Recognition](https://mlsanity.com/paper/2110.14381/) (2021)
- [Hierarchical Compositional Representations for Few-shot Action Recognition](https://mlsanity.com/paper/2208.09424/) (2022)
- [Human-Centric Transformer for Domain Adaptive Action Recognition](https://mlsanity.com/paper/2407.10860/) (2024)
- [CDST: Color Disentangled Style Transfer for Universal Style Reference Customization](https://mlsanity.com/paper/2506.13770/) (2025)
- [RetouchingFFHQ: A Large-scale Dataset for Fine-grained Face Retouching Detection](https://mlsanity.com/paper/2307.10642/) (2023)
- [Video Is Graph: Structured Graph Module for Video Action Recognition](https://mlsanity.com/paper/2110.05904/) (2021)
- [Rain Removal from Light Field Images with 4D Convolution and Multi-scale Gaussian Process](https://mlsanity.com/paper/2208.07735/) (2022)
- [Action Keypoint Network for Efficient Video Recognition](https://mlsanity.com/paper/2201.06304/) (2022)
- [F4D: Factorized 4D Convolutional Neural Network for Efficient Video-level Representation Learning](https://mlsanity.com/paper/2401.08609/) (2023)
- [Hyper-parameter tuning for text guided image editing](https://mlsanity.com/paper/2407.21703/) (2024)
- [A Comprehensive Methodological Survey of Human Activity Recognition Across Divers Data Modalities](https://mlsanity.com/paper/2409.09678/) (2024)
- [GeoDeformer: Geometric Deformable Transformer for Action Recognition](https://mlsanity.com/paper/2311.17975/) (2023)
- [Style-CCL: Content-Preserving Style Transfer via Curriculum Continual Learning](https://mlsanity.com/paper/2606.14746/) (2026)
- [IndGIC: Supervised Action Recognition under Low Illumination](https://mlsanity.com/paper/2308.15345/) (2023)
- [STMixer: A One-Stage Sparse Action Detector](https://mlsanity.com/paper/2404.09842/) (2024)

## Abstract (arXiv)

Most existing 3D CNNs for video representation learning are clip-based methods, and thus do not consider video-level temporal evolution of spatio-temporal features. In this paper, we propose Video-level 4D Convolutional Neural Networks, referred as V4D, to model the evolution of long-range spatio-temporal representation with 4D convolutions, and at the same time, to preserve strong 3D spatio-temporal representation with residual connections. Specifically, we design a new 4D residual block able to capture inter-clip interactions, which could enhance the representation power of the original clip-level 3D CNNs. The 4D residual blocks can be easily integrated into the existing 3D CNNs to perform long-range modeling hierarchically. We further introduce the training and inference methods for the proposed V4D. Extensive experiments are conducted on three video recognition benchmarks, where V4D achieves excellent results, surpassing recent 3D CNNs by a large margin.