# Video Pretraining Advances 3D Deep Learning on Chest CT Tasks

arXiv: https://arxiv.org/abs/2304.00546 · HTML: https://mlsanity.com/paper/2304.00546/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 2 Apr 2023.

## Builds on (most cited first, 21 of 21)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos](https://mlsanity.com/paper/2104.07905/) (2021)
- [CheXtransfer: Performance and Parameter Efficiency of ImageNet Models for Chest X-Ray Interpretation](https://mlsanity.com/paper/2101.06871/) (2021)
- [NiftyNet: a deep-learning platform for medical imaging](https://mlsanity.com/paper/1709.03485/) (2017)
- [Robust and Efficient Medical Imaging with Self-Supervision](https://mlsanity.com/paper/2205.09723/) (2022)
- [RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHR](https://mlsanity.com/paper/2111.11665/) (2021)
- [Improving Computer-aided Detection using Convolutional Neural Networks and Random View Aggregation](https://mlsanity.com/paper/1505.03046/) (2015)
- [Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detection](https://mlsanity.com/paper/2109.07029/) (2021)
- [Deep Learning Based HPV Status Prediction for Oropharyngeal Cancer Patients](https://mlsanity.com/paper/2011.08555/) (2020)
- [ViPTT-Net: Video pretraining of spatio-temporal model for tuberculosis type classification from chest CT scans](https://mlsanity.com/paper/2105.12810/) (2021)

## Built on by (most cited first, 4 of 4)

- [Inter-slice Super-resolution of Magnetic Resonance Images by Pre-training and Self-supervised Fine-tuning](https://mlsanity.com/paper/2406.05974/) (2024)
- [Video and Synthetic MRI Pre-training of 3D Vision Architectures for Neuroimage Analysis](https://mlsanity.com/paper/2309.04651/) (2023)
- [Triage of 3D pathology data via 2.5D multiple-instance learning to guide pathologist assessments](https://mlsanity.com/paper/2406.07061/) (2024)
- [Non-Contrast CT Esophageal Varices Grading through Clinical Prior-Enhanced Multi-Organ Analysis](https://mlsanity.com/paper/2512.19415/) (2025)

## Abstract (arXiv)

Pretraining on large natural image classification datasets such as ImageNet has aided model development on data-scarce 2D medical tasks. 3D medical tasks often have much less data than 2D medical tasks, prompting practitioners to rely on pretrained 2D models to featurize slices. However, these 2D models have been surpassed by 3D models on 3D computer vision benchmarks since they do not natively leverage cross-sectional or temporal information. In this study, we explore whether natural video pretraining for 3D models can enable higher performance on smaller datasets for 3D medical tasks. We demonstrate video pretraining improves the average performance of seven 3D models on two chest CT datasets, regardless of finetuning dataset size, and that video pretraining allows 3D models to outperform 2D baselines. Lastly, we observe that pretraining on the large-scale out-of-domain Kinetics dataset improves performance more than pretraining on a typically-sized in-domain CT dataset. Our results show consistent benefits of video pretraining across a wide array of architectures, tasks, and training dataset sizes, supporting a shift from small-scale in-domain pretraining to large-scale out-of-domain pretraining for 3D medical tasks. Our code is available at: https://github.com/rajpurkarlab/chest-ct-pretraining