# Robustness Testing of Black-Box Models Against CT Degradation Through Test-Time Augmentation

arXiv: https://arxiv.org/abs/2406.19557 · HTML: https://mlsanity.com/paper/2406.19557/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 71 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [CovidGAN: Data Augmentation Using Auxiliary Classifier GAN for Improved Covid-19 Detection](https://mlsanity.com/paper/2103.05094/) (2021)
- [On the use of Mahalanobis distance for out-of-distribution detection with neural networks for medical imaging](https://mlsanity.com/paper/2309.01488/) (2023)
- [ROOD-MRI: Benchmarking the robustness of deep learning segmentation models to out-of-distribution and corrupted data in MRI](https://mlsanity.com/paper/2203.06060/) (2022)
- [Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation](https://mlsanity.com/paper/2306.13528/) (2023)
- [ModelObfuscator: Obfuscating Model Information to Protect Deployed ML-based Systems](https://mlsanity.com/paper/2306.06112/) (2023)
- [On the Out of Distribution Robustness of Foundation Models in Medical Image Segmentation](https://mlsanity.com/paper/2311.11096/) (2023)
- [Redesigning Out-of-Distribution Detection on 3D Medical Images](https://mlsanity.com/paper/2308.07324/) (2023)

## Built on by (most cited first, 1 of 1)

- [Improving Calibration of Black-Box Radiology AI Using Test-Time Augmentation](https://mlsanity.com/paper/2609.29931/) (2026)

## Abstract (arXiv)

Deep learning models for medical image segmentation and object detection are becoming increasingly available as clinical products. However, as details are rarely provided about the training data, models may unexpectedly fail when cases differ from those in the training distribution. An approach allowing potential users to independently test the robustness of a model, treating it as a black box and using only a few cases from their own site, is key for adoption. To address this, a method to test the robustness of these models against CT image quality variation is presented. In this work we present this framework by demonstrating that given the same training data, the model architecture and data pre processing greatly affect the robustness of several frequently used segmentation and object detection methods to simulated CT imaging artifacts and degradation. Our framework also addresses the concern about the sustainability of deep learning models in clinical use, by considering future shifts in image quality due to scanner deterioration or imaging protocol changes which are not reflected in a limited local test dataset.