# Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?

arXiv: https://arxiv.org/abs/2004.04968 · HTML: https://mlsanity.com/paper/2004.04968/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2020.

## Built on by (most cited first, 30 of 31)

- [TCLR: Temporal Contrastive Learning for Video Representation](https://mlsanity.com/paper/2101.07974/) (2021)
- [Skeleton Aware Multi-modal Sign Language Recognition](https://mlsanity.com/paper/2103.08833/) (2021)
- [AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling](https://mlsanity.com/paper/2111.00772/) (2021)
- [Multimodal Distillation for Egocentric Action Recognition](https://mlsanity.com/paper/2307.07483/) (2023)
- [Refining activation downsampling with SoftPool](https://mlsanity.com/paper/2101.00440/) (2021)
- [Learning from Temporal Gradient for Semi-supervised Action Recognition](https://mlsanity.com/paper/2111.13241/) (2021)
- [VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer](https://mlsanity.com/paper/2107.02681/) (2021)
- [Revisiting spatio-temporal layouts for compositional action recognition](https://mlsanity.com/paper/2111.01936/) (2021)
- [Advancing COVID-19 Diagnosis with Privacy-Preserving Collaboration in Artificial Intelligence](https://mlsanity.com/paper/2111.09461/) (2021)
- [Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble](https://mlsanity.com/paper/2110.06161/) (2021)
- [Chop & Learn: Recognizing and Generating Object-State Compositions](https://mlsanity.com/paper/2309.14339/) (2023)
- [Analysis and Extensions of Adversarial Training for Video Classification](https://mlsanity.com/paper/2206.07953/) (2022)
- [Temporal Alignment-Free Video Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2504.05956/) (2025)
- [PIP: Physical Interaction Prediction via Mental Simulation with Span Selection](https://mlsanity.com/paper/2109.04683/) (2021)
- [Improving Transfer Learning with a Dual Image and Video Transformer for Multi-label Movie Trailer Genre Classification](https://mlsanity.com/paper/2210.07983/) (2022)
- [Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows](https://mlsanity.com/paper/2106.08513/) (2021)
- [Unsupervised Action Localization Crop in Video Retargeting for 3D ConvNets](https://mlsanity.com/paper/2111.07426/) (2021)
- [On Negative Sampling for Audio-Visual Contrastive Learning from Movies](https://mlsanity.com/paper/2205.00073/) (2022)
- [Adaptive occlusion sensitivity analysis for visually explaining video recognition networks](https://mlsanity.com/paper/2207.12859/) (2022)
- [QAFE-Net: Quality Assessment of Facial Expressions with Landmark Heatmaps](https://mlsanity.com/paper/2312.00856/) (2023)
- [FireMatch: A Semi-Supervised Video Fire Detection Network Based on Consistency and Distribution Alignment](https://mlsanity.com/paper/2311.05168/) (2023)
- [Measuring Error Alignment for Decision-Making Systems](https://mlsanity.com/paper/2409.13919/) (2024)
- [Physics-Guided Motion Loss for Video Generation Model](https://mlsanity.com/paper/2506.02244/) (2025)
- [Skimming and Scanning for Untrimmed Video Action Recognition](https://mlsanity.com/paper/2104.10492/) (2021)
- [Deep Neural Network approaches for Analysing Videos of Music Performances](https://mlsanity.com/paper/2205.11232/) (2022)
- [Hand Gestures Recognition in Videos Taken with Lensless Camera](https://mlsanity.com/paper/2210.08233/) (2022)
- [Anatomically aware dual-hop learning for pulmonary embolism detection in CT pulmonary angiograms](https://mlsanity.com/paper/2303.17593/) (2023)
- [Subtle Signals: Video-based Detection of Infant Non-nutritive Sucking as a Neurodevelopmental Cue](https://mlsanity.com/paper/2310.16138/) (2023)
- [Traffic Incident Database with Multiple Labels Including Various Perspective Environmental Information](https://mlsanity.com/paper/2312.10737/) (2023)
- [ERDES: A Benchmark Video Dataset for Retinal Detachment and Macular Status Classification in Ocular Ultrasound](https://mlsanity.com/paper/2508.04735/) (2025)

## Abstract (arXiv)

How can we collect and use a video dataset to further improve spatiotemporal 3D Convolutional Neural Networks (3D CNNs)? In order to positively answer this open question in video recognition, we have conducted an exploration study using a couple of large-scale video datasets and 3D CNNs. In the early era of deep neural networks, 2D CNNs have been better than 3D CNNs in the context of video recognition. Recent studies revealed that 3D CNNs can outperform 2D CNNs trained on a large-scale video dataset. However, we heavily rely on architecture exploration instead of dataset consideration. Therefore, in the present paper, we conduct exploration study in order to improve spatiotemporal 3D CNNs as follows: (i) Recently proposed large-scale video datasets help improve spatiotemporal 3D CNNs in terms of video classification accuracy. We reveal that a carefully annotated dataset (e.g., Kinetics-700) effectively pre-trains a video representation for a video classification task. (ii) We confirm the relationships between #category/#instance and video classification accuracy. The results show that #category should initially be fixed, and then #instance is increased on a video dataset in case of dataset construction. (iii) In order to practically extend a video dataset, we simply concatenate publicly available datasets, such as Kinetics-700 and Moments in Time (MiT) datasets. Compared with Kinetics-700 pre-training, we further enhance spatiotemporal 3D CNNs with the merged dataset, e.g., +0.9, +3.4, and +1.1 on UCF-101, HMDB-51, and ActivityNet datasets, respectively, in terms of fine-tuning. (iv) In terms of recognition architecture, the Kinetics-700 and merged dataset pre-trained models increase the recognition performance to 200 layers with the Residual Network (ResNet), while the Kinetics-400 pre-trained model cannot successfully optimize the 200-layer architecture.