The Confidence Gate Theorem: When Should Ranked Decision Systems Abstain?
Not on the map. Why?
Abstract (arXiv)
Ranked decision systems -- recommenders, ad auctions, clinical triage queues -- must decide when to intervene in ranked outputs and when to abstain. We study when confidence-based abstention monotonically improves decision quality, and when it fails. The formal conditions are simple: rank-alignment and no inversion zones. The substantive contribution is identifying why these conditions hold or fail: the distinction between structural uncertainty (missing data, e.g., cold-start) and contextual uncertainty (missing context, e.g., temporal drift). Empirically, we validate this distinction across three domains: collaborative filtering (MovieLens, 3 distribution shifts), e-commerce intent detection (RetailRocket, Criteo, Yoochoose), and clinical pathway triage (MIMIC-IV). Structural uncertainty produces near-monotonic abstention gains in all domains; structurally grounded confidence signals (observation counts) fail under contextual drift, producing as many monotonicity violations as random abstention on our MovieLens temporal split. Context-aware alternatives -- ensemble disagreement and recency features -- substantially narrow the gap (reducing violations from 3 to 1--2) but do not fully restore monotonicity, suggesting that contextual uncertainty poses qualitatively different challenges. Exception labels defined from residuals degrade substantially under distribution shift (AUC drops from 0.71 to 0.61--0.62 across three splits), providing a clean negative result against the common practice of exception-based intervention. The results provide a practical deployment diagnostic: check C1 and C2 on held-out data before deploying a confidence gate, and match the confidence signal to the dominant uncertainty type.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 45 references that are in the map. First posted on arXiv 10 Mar 2026.
Builds on
26 of 45 references.
References · 26
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2018Deep Anomaly Detection with Outlier Exposurecited by 942
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
- 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
- 2022Conformal Risk Controlcited by 299
- 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
14 more
- 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
- 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
- 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
- 2023Improved Online Conformal Prediction via Strongly Adaptive Online Learningcited by 78
- 2023Who Should Predict? Exact Algorithms For Learning to Defer to Humanscited by 70
- 2022Calibrated Learning to Defer with One-vs-All Classifierscited by 69
- 2022Selection by Prediction with Conformal p-valuescited by 69
- 2021Optimal strategies for reject option classifierscited by 61
- 2022Sample Efficient Learning of Predictors that Complement Humanscited by 46
- 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 37
- 2023Non-Exchangeable Conformal Risk Controlcited by 29
- 2022Recommendation Systems with Distribution-Free Reliability Guaranteescited by 13
- 2024Hierarchical Selective Classificationcited by 7
- 2017The Relationship Between Agnostic Selective Classification Active Learning and the Disagreement Coefficientcited by 6
Built on by
Not cited yet.