# Deep learning-based COVID-19 pneumonia classification using chest CT images: model generalizability

arXiv: https://arxiv.org/abs/2102.09616 · HTML: https://mlsanity.com/paper/2102.09616/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 45 references that are in the map.
- First posted on arXiv 18 Feb 2021.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [MosMedData: Chest CT Scans With COVID-19 Related Findings Dataset](https://mlsanity.com/paper/2005.06465/) (2020)
- [Covid-19: Automatic detection from X-Ray images utilizing Transfer Learning with Convolutional Neural Networks](https://mlsanity.com/paper/2003.11617/) (2020)
- [Automatic Detection of Coronavirus Disease (COVID-19) Using X-ray Images and Deep Convolutional Neural Networks](https://mlsanity.com/paper/2003.10849/) (2020)
- [Deep Learning COVID-19 Features on CXR using Limited Training Data Sets](https://mlsanity.com/paper/2004.05758/) (2020)
- [COVIDX-Net: A Framework of Deep Learning Classifiers to Diagnose COVID-19 in X-Ray Images](https://mlsanity.com/paper/2003.11055/) (2020)
- [Contrastive Cross-site Learning with Redesigned Net for COVID-19 CT Classification](https://mlsanity.com/paper/2009.07652/) (2020)

## Built on by (most cited first, 5 of 5)

- [The Utility of the Virtual Imaging Trials Methodology for Objective Characterization of AI Systems and Training Data](https://mlsanity.com/paper/2308.09730/) (2023)
- [Self-supervised Model Based on Masked Autoencoders Advance CT Scans Classification](https://mlsanity.com/paper/2210.05073/) (2022)
- [Performance Deterioration of Deep Learning Models after Clinical Deployment: A Case Study with Auto-segmentation for Definitive Prostate Cancer Radiotherapy](https://mlsanity.com/paper/2210.05673/) (2022)
- [Explainable and Lightweight Model for COVID-19 Detection Using Chest Radiology Images](https://mlsanity.com/paper/2212.13788/) (2022)
- [Evaluating Generalizability of Deep Learning Models Using Indian-COVID-19 CT Dataset](https://mlsanity.com/paper/2212.13929/) (2022)

## Abstract (arXiv)

Since the outbreak of the COVID-19 pandemic, worldwide research efforts have focused on using artificial intelligence (AI) technologies on various medical data of COVID-19-positive patients in order to identify or classify various aspects of the disease, with promising reported results. However, concerns have been raised over their generalizability, given the heterogeneous factors in training datasets. This study aims to examine the severity of this problem by evaluating deep learning (DL) classification models trained to identify COVID-19-positive patients on 3D computed tomography (CT) datasets from different countries. We collected one dataset at UT Southwestern (UTSW), and three external datasets from different countries: CC-CCII Dataset (China), COVID-CTset (Iran), and MosMedData (Russia). We divided the data into 2 classes: COVID-19-positive and COVID-19-negative patients. We trained nine identical DL-based classification models by using combinations of the datasets with a 72% train, 8% validation, and 20% test data split. The models trained on a single dataset achieved accuracy/area under the receiver operating characteristics curve (AUC) values of 0.87/0.826 (UTSW), 0.97/0.988 (CC-CCCI), and 0.86/0.873 (COVID-CTset) when evaluated on their own dataset. The models trained on multiple datasets and evaluated on a test set from one of the datasets used for training performed better. However, the performance dropped close to an AUC of 0.5 (random guess) for all models when evaluated on a different dataset outside of its training datasets. Including the MosMedData, which only contained positive labels, into the training did not necessarily help the performance on the other datasets. Multiple factors likely contribute to these results, such as patient demographics and differences in image acquisition or reconstruction, causing a data shift among different study cohorts.