# Optimal strategies for reject option classifiers

arXiv: https://arxiv.org/abs/2101.12523 · HTML: https://mlsanity.com/paper/2101.12523/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 42 references that are in the map.
- First posted on arXiv 29 Jan 2021.

## Builds on (most cited first, 4 of 4)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)

## Built on by (most cited first, 30 of 61)

- [Machine Learning with a Reject Option: A survey](https://mlsanity.com/paper/2107.11277/) (2021)
- [Is Out-of-Distribution Detection Learnable?](https://mlsanity.com/paper/2210.14707/) (2022)
- [Overcoming Common Flaws in the Evaluation of Selective Classification Systems](https://mlsanity.com/paper/2407.01032/) (2024)
- [How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks](https://mlsanity.com/paper/2305.15508/) (2023)
- [A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators](https://mlsanity.com/paper/2410.15361/) (2024)
- [A Causal Framework for Evaluating Deferring Systems](https://mlsanity.com/paper/2405.18902/) (2024)
- [Selective Classification Under Distribution Shifts](https://mlsanity.com/paper/2405.05160/) (2024)
- [Selective Conformal Risk Control](https://mlsanity.com/paper/2512.12844/) (2025)
- [Revisiting Confidence Estimation: Towards Reliable Failure Prediction](https://mlsanity.com/paper/2403.02886/) (2024)
- [Moment of Untruth: Dealing with Negative Queries in Video Moment Retrieval](https://mlsanity.com/paper/2502.08544/) (2025)
- [Confident AI](https://mlsanity.com/paper/2202.05957/) (2022)
- [To Ask or Not to Ask: Learning to Require Human Feedback](https://mlsanity.com/paper/2510.08314/) (2025)
- [How Far Can Fairness Constraints Help Recover From Biased Data?](https://mlsanity.com/paper/2312.10396/) (2023)
- [Classification with Conceptual Safeguards](https://mlsanity.com/paper/2411.04342/) (2024)
- [MCQA-Eval: Efficient Confidence Evaluation in NLG with Gold-Standard Correctness Labels](https://mlsanity.com/paper/2502.14268/) (2025)
- [When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation](https://mlsanity.com/paper/2606.29054/) (2026)
- [SCOD: From Heuristics to Theory](https://mlsanity.com/paper/2403.16916/) (2024)
- [Rejection via Learning Density Ratios](https://mlsanity.com/paper/2405.18686/) (2024)
- [Classification with Reject Option: Distribution-free Error Guarantees via Conformal Prediction](https://mlsanity.com/paper/2506.21802/) (2025)
- [Explicit Abstention Knobs for Predictable Reliability in Video Question Answering](https://mlsanity.com/paper/2601.00138/) (2025)
- [Reject option models comprising out-of-distribution detection](https://mlsanity.com/paper/2307.05199/) (2023)
- [Interpretable and Fair Mechanisms for Abstaining Classifiers](https://mlsanity.com/paper/2503.18826/) (2025)
- [ALARM: Automated MLLM-Based Anomaly Detection in Complex-EnviRonment Monitoring with Uncertainty Quantification](https://mlsanity.com/paper/2512.03101/) (2025)
- [Aligning Language Models with Selective Prediction](https://mlsanity.com/paper/2607.03528/) (2026)
- [Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learning](https://mlsanity.com/paper/2607.14817/) (2026)
- [Precision and Recall Reject Curves for Classification](https://mlsanity.com/paper/2308.08381/) (2023)
- [Soft Dice Confidence: A Near-Optimal Confidence Estimator for Selective Prediction in Semantic Segmentation](https://mlsanity.com/paper/2402.10665/) (2024)
- [Reweighting Improves Conditional Risk Bounds](https://mlsanity.com/paper/2501.02353/) (2025)
- [Regretful Decisions under Label Noise](https://mlsanity.com/paper/2504.09330/) (2025)
- [A Connection Between Learning to Reject and Bhattacharyya Divergences](https://mlsanity.com/paper/2505.05273/) (2025)

## Abstract (arXiv)

In classification with a reject option, the classifier is allowed in uncertain cases to abstain from prediction. The classical cost-based model of a reject option classifier requires the cost of rejection to be defined explicitly. An alternative bounded-improvement model, avoiding the notion of the reject cost, seeks for a classifier with a guaranteed selective risk and maximal cover. We coin a symmetric definition, the bounded-coverage model, which seeks for a classifier with minimal selective risk and guaranteed coverage. We prove that despite their different formulations the three rejection models lead to the same prediction strategy: a Bayes classifier endowed with a randomized Bayes selection function. We define a notion of a proper uncertainty score as a scalar summary of prediction uncertainty sufficient to construct the randomized Bayes selection function. We propose two algorithms to learn the proper uncertainty score from examples for an arbitrary black-box classifier. We prove that both algorithms provide Fisher consistent estimates of the proper uncertainty score and we demonstrate their efficiency on different prediction problems including classification, ordinal regression and structured output classification.