# ViPTT-Net: Video pretraining of spatio-temporal model for tuberculosis type classification from chest CT scans

arXiv: https://arxiv.org/abs/2105.12810 · HTML: https://mlsanity.com/paper/2105.12810/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 26 May 2021.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Recent Advances in Recurrent Neural Networks](https://mlsanity.com/paper/1801.01078/) (2017)
- [Uniformizing Techniques to Process CT scans with 3D CNNs for Tuberculosis Prediction](https://mlsanity.com/paper/2007.13224/) (2020)
- [RADNET: Radiologist Level Accuracy using Deep Learning for HEMORRHAGE detection in CT Scans](https://mlsanity.com/paper/1710.04934/) (2017)
- [Melanoma Detection using Adversarial Training and Deep Transfer Learning](https://mlsanity.com/paper/2004.06824/) (2020)
- [Deep Sequential Learning for Cervical Spine Fracture Detection on Computed Tomography Imaging](https://mlsanity.com/paper/2010.13336/) (2020)
- [A Deep Learning System That Generates Quantitative CT Reports for Diagnosing Pulmonary Tuberculosis](https://mlsanity.com/paper/1910.02285/) (2019)

## Built on by (most cited first, 2 of 2)

- [Video Pretraining Advances 3D Deep Learning on Chest CT Tasks](https://mlsanity.com/paper/2304.00546/) (2023)
- [EXACT-CT: EXplainable Analysis for Crohn's and Tuberculosis using CT](https://mlsanity.com/paper/2503.00159/) (2025)

## Abstract (arXiv)

Pretraining has sparked groundswell of interest in deep learning workflows to learn from limited data and improve generalization. While this is common for 2D image classification tasks, its application to 3D medical imaging tasks like chest CT interpretation is limited. We explore the idea of whether pretraining a model on realistic videos could improve performance rather than training the model from scratch, intended for tuberculosis type classification from chest CT scans. To incorporate both spatial and temporal features, we develop a hybrid convolutional neural network (CNN) and recurrent neural network (RNN) model, where the features are extracted from each axial slice of the CT scan by a CNN, these sequence of image features are input to a RNN for classification of the CT scan. Our model termed as ViPTT-Net, was trained on over 1300 video clips with labels of human activities, and then fine-tuned on chest CT scans with labels of tuberculosis type. We find that pretraining the model on videos lead to better representations and significantly improved model validation performance from a kappa score of 0.17 to 0.35, especially for under-represented class samples. Our best method achieved 2nd place in the ImageCLEF 2021 Tuberculosis - TBT classification task with a kappa score of 0.20 on the final test set with only image information (without using clinical meta-data). All codes and models are made available.