# The Utility of the Virtual Imaging Trials Methodology for Objective Characterization of AI Systems and Training Data

arXiv: https://arxiv.org/abs/2308.09730 · HTML: https://mlsanity.com/paper/2308.09730/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 68 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans](https://mlsanity.com/paper/2008.06388/) (2020)
- [CheXclusion: Fairness gaps in deep chest X-ray classifiers](https://mlsanity.com/paper/2003.00827/) (2020)
- [BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patients](https://mlsanity.com/paper/2006.01174/) (2020)
- [COVIDNet-CT: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest CT Images](https://mlsanity.com/paper/2009.05383/) (2020)
- [COVID-CAPS: A Capsule Network-based Framework for Identification of COVID-19 cases from X-ray Images](https://mlsanity.com/paper/2004.02696/) (2020)
- [Classification of Multiple Diseases on Body CT Scans using Weakly Supervised Deep Learning](https://mlsanity.com/paper/2008.01158/) (2020)
- [Deep learning-based COVID-19 pneumonia classification using chest CT images: model generalizability](https://mlsanity.com/paper/2102.09616/) (2021)
- [Virtual vs. Reality: External Validation of COVID-19 Classifiers using XCAT Phantoms for Chest Computed Tomography](https://mlsanity.com/paper/2203.03074/) (2022)
- [Co-occurring Diseases Heavily Influence the Performance of Weakly Supervised Learning Models for Classification of Chest CT](https://mlsanity.com/paper/2202.11709/) (2022)

## Built on by (most cited first, 4 of 4)

- [Virtual Patients, Real Gains: Digital Twin-Based Simulated CT for Multitask Lung Nodule Analysis](https://mlsanity.com/paper/2502.21187/) (2025)
- [iTRIALSPACE: Programmable Virtual Lesion Trials for Controlled Evaluation of Lung CT Models](https://mlsanity.com/paper/2605.05761/) (2026)
- [EXACT-CT: EXplainable Analysis for Crohn's and Tuberculosis using CT](https://mlsanity.com/paper/2503.00159/) (2025)
- [When Does Synthetic CT Transfer? A Label-Free Donor/Host Diagnostic for Medical Vision-Language Model Routing on Real Lung CT](https://mlsanity.com/paper/2606.29232/) (2026)

## Abstract (arXiv)

Purpose: The credibility of Artificial Intelligence (AI) models for medical imaging continues to be a challenge, affected by the diversity of models, the data used to train the models, and applicability of their combination to produce reproducible results for new data. In this work, we aimed to explore whether emerging Virtual Imaging Trials (VIT) methodologies can provide an objective resource to approach this challenge. Approach: The study was conducted for the case example of COVID-19 diagnosis using clinical and virtual computed tomography (CT) and chest radiography (CXR) processed with convolutional neural networks. Multiple AI models were developed and tested using 3D ResNet-like and 2D EfficientNetv2 architectures across diverse datasets. Results: Model performance was evaluated using the area under the curve (AUC) and the DeLong method for AUC confidence intervals. The models trained on the most diverse datasets showed the highest external testing performance, with AUC values ranging from 0.73-0.76 for CT and 0.70-0.73 for CXR. Internal testing yielded higher AUC values (0.77-0.85 for CT and 0.77-1.0 for CXR), highlighting a substantial drop in performance during external validation, which underscores the importance of diverse and comprehensive training and testing data. Most notably, the VIT approach provided objective assessment of the utility of diverse models and datasets, while offering insight into the influence of dataset characteristics, patient factors, and imaging physics on AI efficacy. Conclusions: The VIT approach enhances model transparency and reliability, offering nuanced insights into the factors driving AI performance and bridging the gap between experimental and clinical settings.