# Performance evaluation of predictive AI models to support medical decisions: Overview and guidance

arXiv: https://arxiv.org/abs/2412.10288 · HTML: https://mlsanity.com/paper/2412.10288/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2024.

## Built on by (most cited first, 9 of 9)

- [A Consensus Privacy Metrics Framework for Synthetic Data](https://mlsanity.com/paper/2503.04980/) (2025)
- [Synthetic Data Generation for Augmenting Small Samples](https://mlsanity.com/paper/2501.18741/) (2025)
- [Critical Appraisal of Fairness Metrics in Clinical Predictive AI](https://mlsanity.com/paper/2506.17035/) (2025)
- [VOLMO: Versatile and Open Large Models for Ophthalmology](https://mlsanity.com/paper/2603.23953/) (2026)
- [A multimodal Bayesian Network for symptom-level depression and anxiety prediction from voice and speech data](https://mlsanity.com/paper/2512.07741/) (2025)
- [Explainable and externally validated machine learning for neurocognitive diagnosis via electrocardiograms](https://mlsanity.com/paper/2502.04918/) (2025)
- [Evaluating classification performance across operating contexts: A comparison of decision curve analysis and cost curves](https://mlsanity.com/paper/2509.24608/) (2025)
- [Machine learning models for estimating counterfactuals in a single-arm inflammatory bowel disease study](https://mlsanity.com/paper/2604.23465/) (2026)
- [Risk Is Not the Target: A Monotonic Framework for Evaluating Wildfire Operational Risk Signals](https://mlsanity.com/paper/2607.21597/) (2026)

## Abstract (arXiv)

A myriad of measures to illustrate performance of predictive artificial intelligence (AI) models have been proposed in the literature. Selecting appropriate performance measures is essential for predictive AI models that are developed to be used in medical practice, because poorly performing models may harm patients and lead to increased costs. We aim to assess the merits of classic and contemporary performance measures when validating predictive AI models for use in medical practice. We focus on models with a binary outcome. We discuss 32 performance measures covering five performance domains (discrimination, calibration, overall, classification, and clinical utility) along with accompanying graphical assessments. The first four domains cover statistical performance, the fifth domain covers decision-analytic performance. We explain why two key characteristics are important when selecting which performance measures to assess: (1) whether the measure's expected value is optimized when it is calculated using the correct probabilities (i.e., a "proper" measure), and (2) whether they reflect either purely statistical performance or decision-analytic performance by properly considering misclassification costs. Seventeen measures exhibit both characteristics, fourteen measures exhibited one characteristic, and one measure possessed neither characteristic (the F1 measure). All classification measures (such as classification accuracy and F1) are improper for clinically relevant decision thresholds other than 0.5 or the prevalence. We recommend the following measures and plots as essential to report: AUROC, calibration plot, a clinical utility measure such as net benefit with decision curve analysis, and a plot with probability distributions per outcome category.