# Collapsibility of Performance Metrics in Clinical Predictive AI

arXiv: https://arxiv.org/abs/2608.30568 · HTML: https://mlsanity.com/paper/2608.30568/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 31 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Builds on (most cited first, 7 of 7)

- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the xAUC Metric](https://mlsanity.com/paper/1902.05826/) (2019)
- [Critical Appraisal of Fairness Metrics in Clinical Predictive AI](https://mlsanity.com/paper/2506.17035/) (2025)
- [On the collapsibility of measures of effect in the counterfactual causal framework](https://mlsanity.com/paper/1610.00033/) (2016)
- [Model-based ROC (mROC) curve: examining the effect of case-mix and model calibration on the ROC plot](https://mlsanity.com/paper/2003.00316/) (2020)
- [A decomposition of Fisher's information to inform sample size for developing or updating fair and precise clinical prediction models -- Part 3: continuous outcomes](https://mlsanity.com/paper/2507.23548/) (2025)
- [Decomposing Global AUC into Cluster-Level Contributions for Localized Model Diagnostics](https://mlsanity.com/paper/2508.07495/) (2025)

## Abstract (arXiv)

Background: Population level assessments of predictive artificial intelligence (AI) can conceal performance disparities across subgroups. Fairness evaluations commonly rely on performance analyses across subgroups. However, some performance metrics are non-collapsible, meaning that the overall population performance value does not equal the weighted average of subgroup specific values. Objective: To examine the collapsibility properties of commonly reported performance metrics in predictive AI, with a focus on the area under the receiver operating characteristic curve (AUC, also known as c-statistic). Methods: We investigate the collapsibility of 15 performance metrics, either by expressing each metric as a linear combination of its stratum specific values or, where non-collapsible, by providing a counterexample inspired by Simpson's paradox as a formal disproof. Results: Five performance metrics (AUC, calibration intercept, calibration slope, expected calibration error, and Nagelkerke R^2) are shown to be non-collapsible, and ten (O:E ratio, logloss, Brier score, accuracy, F1-score, true positive rate, true negative rate, positive predictive value, negative predictive value, and net benefit) are shown to be collapsible. The AUC is shown to be non-collapsible because it decomposes into within- and cross-group AUC terms when subpopulations coexist, such that its overall value may fall outside the range of subgroup specific AUCs. Conclusions: Non-collapsibility of performance metrics has important consequences for reporting, model appraisal, and fairness evaluation. It can generate spurious differences between subgroup and overall performance, which may mislead fairness evaluations. Explicitly acknowledging and reporting the collapsibility properties of performance metrics improves both the interpretability and transparency of fairness assessments.