# Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets

arXiv: https://arxiv.org/abs/2207.02238 · HTML: https://mlsanity.com/paper/2207.02238/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 20 of 20)

- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [Least Ambiguous Set-Valued Classifiers with Bounded Error Levels](https://mlsanity.com/paper/1609.00451/) (2016)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging](https://mlsanity.com/paper/2202.05265/) (2022)
- [Nested conformal prediction and quantile out-of-bag ensemble methods](https://mlsanity.com/paper/1910.10562/) (2019)
- [A Conformal Prediction Approach to Explore Functional Data](https://mlsanity.com/paper/1302.6452/) (2013)
- [Conformal Prediction using Conditional Histograms](https://mlsanity.com/paper/2105.08747/) (2021)
- [Fair Conformal Predictors for Applications in Medical Imaging](https://mlsanity.com/paper/2109.04392/) (2021)
- [Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1909.09884/) (2019)
- [Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design](https://mlsanity.com/paper/2202.03613/) (2022)
- [Private Prediction Sets](https://mlsanity.com/paper/2102.06202/) (2021)
- [Three Applications of Conformal Prediction for Rating Breast Density in Mammography](https://mlsanity.com/paper/2206.12008/) (2022)
- [DeepSPINE: Automated Lumbar Vertebral Segmentation, Disc-level Designation, and Spinal Stenosis Grading Using Deep Learning](https://mlsanity.com/paper/1807.10215/) (2018)

## Built on by (most cited first, 30 of 38)

- [Conformal Prediction with Large Language Models for Multi-Choice Question Answering](https://mlsanity.com/paper/2305.18404/) (2023)
- [Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis](https://mlsanity.com/paper/2210.03736/) (2022)
- [Federated Conformal Predictors for Distributed Uncertainty Quantification](https://mlsanity.com/paper/2305.17564/) (2023)
- [Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction](https://mlsanity.com/paper/2509.18658/) (2025)
- [Conformal Prediction: A Data Perspective](https://mlsanity.com/paper/2410.06494/) (2024)
- [An Information Theoretic Perspective on Conformal Prediction](https://mlsanity.com/paper/2405.02140/) (2024)
- [Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration](https://mlsanity.com/paper/2406.06818/) (2024)
- [Conformal Prediction via Regression-as-Classification](https://mlsanity.com/paper/2404.08168/) (2024)
- [A Conformal Prediction Score that is Robust to Label Noise](https://mlsanity.com/paper/2405.02648/) (2024)
- [Empirically Validating Conformal Prediction on Modern Vision Architectures Under Distribution Shift and Long-tailed Data](https://mlsanity.com/paper/2307.01088/) (2023)
- [Conformal Risk Control for Ordinal Classification](https://mlsanity.com/paper/2405.00417/) (2024)
- [Task-Driven Uncertainty Quantification in Inverse Problems via Conformal Prediction](https://mlsanity.com/paper/2405.18527/) (2024)
- [Direct Prediction Set Minimization via Bilevel Conformal Classifier Training](https://mlsanity.com/paper/2506.06599/) (2025)
- [Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Prediction](https://mlsanity.com/paper/2506.17503/) (2025)
- [Quantile Learn-Then-Test: Quantile-Based Risk Control for Hyperparameter Optimization](https://mlsanity.com/paper/2407.17358/) (2024)
- [Pitfalls of Conformal Predictions for Medical Image Classification](https://mlsanity.com/paper/2506.18162/) (2025)
- [Conformalized Selective Regression](https://mlsanity.com/paper/2402.16300/) (2024)
- [Conformal Prediction of Classifiers with Many Classes based on Noisy Labels](https://mlsanity.com/paper/2501.12749/) (2025)
- [WQLCP: Weighted Adaptive Conformal Prediction for Robust Uncertainty Quantification Under Distribution Shifts](https://mlsanity.com/paper/2505.19587/) (2025)
- [Reliable Multi-View Learning with Conformal Prediction for Aortic Stenosis Classification in Echocardiography](https://mlsanity.com/paper/2409.09680/) (2024)
- [RR-CP: Reliable-Region-Based Conformal Prediction for Trustworthy Medical Image Classification](https://mlsanity.com/paper/2309.04760/) (2023)
- [Distribution-free Conformal Prediction for Ordinal Classification](https://mlsanity.com/paper/2404.16610/) (2024)
- [Conformal Depression Prediction](https://mlsanity.com/paper/2405.18723/) (2024)
- [Adaptive Conformal Guidance for Learning under Uncertainty](https://mlsanity.com/paper/2502.16736/) (2025)
- [Privacy-Preserving Conformal Prediction Under Local Differential Privacy](https://mlsanity.com/paper/2505.15721/) (2025)
- [A conformalized learning of a prediction set with applications to medical imaging classification](https://mlsanity.com/paper/2408.05037/) (2024)
- [Efficient Conformal Prediction for Regression Models under Label Noise](https://mlsanity.com/paper/2509.15120/) (2025)
- [Provably Minimum-Length Conformal Prediction Sets for Ordinal Classification](https://mlsanity.com/paper/2511.16845/) (2025)
- [Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score](https://mlsanity.com/paper/2606.24959/) (2026)
- [Conformalized High-Density Quantile Regression via Dynamic Prototypes-based Probability Density Estimation](https://mlsanity.com/paper/2411.01266/) (2024)

## Abstract (arXiv)

The regulatory approval and broad clinical deployment of medical AI have been hampered by the perception that deep learning models fail in unpredictable and possibly catastrophic ways. A lack of statistically rigorous uncertainty quantification is a significant factor undermining trust in AI results. Recent developments in distribution-free uncertainty quantification present practical solutions for these issues by providing reliability guarantees for black-box models on arbitrary data distributions as formally valid finite-sample prediction intervals. Our work applies these new uncertainty quantification methods -- specifically conformal prediction -- to a deep-learning model for grading the severity of spinal stenosis in lumbar spine MRI. We demonstrate a technique for forming ordinal prediction sets that are guaranteed to contain the correct stenosis severity within a user-defined probability (confidence interval). On a dataset of 409 MRI exams processed by the deep-learning model, the conformal method provides tight coverage with small prediction set sizes. Furthermore, we explore the potential clinical applicability of flagging cases with high uncertainty predictions (large prediction sets) by quantifying an increase in the prevalence of significant imaging abnormalities (e.g. motion artifacts, metallic artifacts, and tumors) that could degrade confidence in predictive performance when compared to a random sample of cases.