# Performance Deterioration of Deep Learning Models after Clinical Deployment: A Case Study with Auto-segmentation for Definitive Prostate Cancer Radiotherapy

arXiv: https://arxiv.org/abs/2210.05673 · HTML: https://mlsanity.com/paper/2210.05673/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 47 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 11 of 11)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Generalization in Deep Learning](https://mlsanity.com/paper/1710.05468/) (2017)
- [NeurIPS 2020 Competition: Predicting Generalization in Deep Learning](https://mlsanity.com/paper/2012.07976/) (2020)
- [Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks](https://mlsanity.com/paper/1908.00690/) (2019)
- [Improving the generalizability of convolutional neural network-based segmentation on CMR images](https://mlsanity.com/paper/1907.01268/) (2019)
- [CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting](https://mlsanity.com/paper/2002.11379/) (2020)
- [Generalizability issues with deep learning models in medicine and their potential solutions: illustrated with Cone-Beam Computed Tomography (CBCT) to Computed Tomography (CT) image conversion](https://mlsanity.com/paper/2004.07700/) (2020)
- [Deep learning-based COVID-19 pneumonia classification using chest CT images: model generalizability](https://mlsanity.com/paper/2102.09616/) (2021)
- [Segmentation by Test-Time Optimization (TTO) for CBCT-based Adaptive Radiation Therapy](https://mlsanity.com/paper/2202.03978/) (2022)
- [A Proof-of-Concept Study of Artificial Intelligence Assisted Contour Revision](https://mlsanity.com/paper/2107.13465/) (2021)

## Built on by (most cited first, 2 of 2)

- [Doctor-R1: Mastering Clinical Inquiry with Experiential Agentic Reinforcement Learning](https://mlsanity.com/paper/2510.04284/) (2025)
- [A Tutorial on Clinical Speech AI Development: From Data Collection to Model Validation](https://mlsanity.com/paper/2410.21640/) (2024)

## Abstract (arXiv)

We evaluated the temporal performance of a deep learning (DL) based artificial intelligence (AI) model for auto segmentation in prostate radiotherapy, seeking to correlate its efficacy with changes in clinical landscapes. Our study involved 1328 prostate cancer patients who underwent definitive radiotherapy from January 2006 to August 2022 at the University of Texas Southwestern Medical Center. We trained a UNet based segmentation model on data from 2006 to 2011 and tested it on data from 2012 to 2022 to simulate real world clinical deployment. We measured the model performance using the Dice similarity coefficient (DSC), visualized the trends in contour quality using exponentially weighted moving average (EMA) curves. Additionally, we performed Wilcoxon Rank Sum Test to analyze the differences in DSC distributions across distinct periods, and multiple linear regression to investigate the impact of various clinical factors. The model exhibited peak performance in the initial phase (from 2012 to 2014) for segmenting the prostate, rectum, and bladder. However, we observed a notable decline in performance for the prostate and rectum after 2015, while bladder contour quality remained stable. Key factors that impacted the prostate contour quality included physician contouring styles, the use of various hydrogel spacer, CT scan slice thickness, MRI-guided contouring, and using intravenous (IV) contrast. Rectum contour quality was influenced by factors such as slice thickness, physician contouring styles, and the use of various hydrogel spacers. The bladder contour quality was primarily affected by using IV contrast. This study highlights the challenges in maintaining AI model performance consistency in a dynamic clinical setting. It underscores the need for continuous monitoring and updating of AI models to ensure their ongoing effectiveness and relevance in patient care.