mlsanityData to 14 Oct 2026

A Deployment Audit of Release-Side Risk in Conformal Triage under Prevalence Shift

arXiv 2605.20956 ↗First posted 20 May 2026cs.LG cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

Conformal triage converts predictive scores into deployment actions that either release a case, flag it for urgent attention, or defer it to human review. Under an observed change in target-event prevalence, however, marginal coverage and human-review rate can miss whether patients who experience the target event are released without review. To address this gap, we introduce a leakage-aware deployment audit for release-side conformal triage. It first assigns target subjects to three non-overlapping roles: prevalence correction, conformal calibration, and held-out release-side evaluation. This separation then lets the audit evaluate release directly: how many event-positive patients are cleared without review, whether the pilot has enough event labels for calibration, and how the release-review trade-off shifts. Applying this audit to a retrospective non-small-cell lung cancer (NSCLC) target cohort shows why lower review can be misleading: after prevalence correction, the pooled conformal branch lowers review by releasing more patients, some of whom are event-positive. Within the audit, the classwise branch acts as a scarcity diagnostic: the pilot has too few event labels to support a low-review release rule.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

21 of 34 references.

References · 21

  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  3. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  4. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  5. 2017Selective Classification for Deep Neural Networkscited by 852
  6. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  7. 2019Conformal Prediction Under Covariate Shiftcited by 519
  8. 2020Classification with Valid and Adaptive Coveragecited by 396
  9. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 366
  10. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  11. 2022Conformal Risk Controlcited by 299
  12. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
9 more
  1. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 272
  2. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  3. 2023Conformal Prediction With Conditional Guaranteescited by 152
  4. 2023Class-Conditional Conformal Prediction with Many Classescited by 109
  5. 2021Distribution-free uncertainty quantification for classification under label shiftcited by 101
  6. 2021Fair Conformal Predictors for Applications in Medical Imagingcited by 74
  7. 2025Out-of-Distribution Detection Methods Answer the Wrong Questionscited by 14
  8. 2023Deployment of Image Analysis Algorithms under Prevalence Shiftscited by 4
  9. 2025Classification with Reject Option: Distribution-free Error Guarantees via Conformal Predictioncited by 3

Built on by

Not cited yet.