# On the Calibration of Multiclass Classification with Rejection

arXiv: https://arxiv.org/abs/1901.10655 · HTML: https://mlsanity.com/paper/1901.10655/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2019.

## Built on by (most cited first, 30 of 41)

- [Machine Learning with a Reject Option: A survey](https://mlsanity.com/paper/2107.11277/) (2021)
- [Is Out-of-Distribution Detection Learnable?](https://mlsanity.com/paper/2210.14707/) (2022)
- [Calibrated Learning to Defer with One-vs-All Classifiers](https://mlsanity.com/paper/2202.03673/) (2022)
- [Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles](https://mlsanity.com/paper/2210.16955/) (2022)
- [When Does Confidence-Based Cascade Deferral Suffice?](https://mlsanity.com/paper/2307.02764/) (2023)
- [Predictor-Rejector Multi-Class Abstention: Theoretical Analysis and Algorithms](https://mlsanity.com/paper/2310.14772/) (2023)
- [Calibrated Selective Classification](https://mlsanity.com/paper/2208.12084/) (2022)
- [Learning to Reject with a Fixed Predictor: Application to Decontextualization](https://mlsanity.com/paper/2301.09044/) (2023)
- [Set-valued classification -- overview via a unified framework](https://mlsanity.com/paper/2102.12318/) (2021)
- [Learning to Defer to a Population: A Meta-Learning Approach](https://mlsanity.com/paper/2403.02683/) (2024)
- [In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defer](https://mlsanity.com/paper/2311.01106/) (2023)
- [Realizable H-Consistent and Bayes-Consistent Loss Functions for Learning to Defer](https://mlsanity.com/paper/2407.13732/) (2024)
- [A Universal Growth Rate for Learning with Smooth Surrogate Losses](https://mlsanity.com/paper/2405.05968/) (2024)
- [Enhanced H-Consistency Bounds](https://mlsanity.com/paper/2407.13722/) (2024)
- [Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defer](https://mlsanity.com/paper/2506.20650/) (2025)
- [Surrogate Regret Bounds for Polyhedral Losses](https://mlsanity.com/paper/2110.14031/) (2021)
- [Plugin estimators for selective classification with out-of-distribution detection](https://mlsanity.com/paper/2301.12386/) (2023)
- [Budgeted Multiple-Expert Deferral](https://mlsanity.com/paper/2510.26706/) (2025)
- [Regression with Cost-based Rejection](https://mlsanity.com/paper/2311.04550/) (2023)
- [When in Doubt, Summon the Titans: Efficient Inference with Large Models](https://mlsanity.com/paper/2110.10305/) (2021)
- [Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Model](https://mlsanity.com/paper/2405.12081/) (2024)
- [From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictions](https://mlsanity.com/paper/2301.11797/) (2023)
- [Online Decision Mediation](https://mlsanity.com/paper/2310.18601/) (2023)
- [How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond](https://mlsanity.com/paper/2501.05714/) (2025)
- [On the Fly Neural Style Smoothing for Risk-Averse Domain Generalization](https://mlsanity.com/paper/2307.08551/) (2023)
- [Partial-Label Learning with a Reject Option](https://mlsanity.com/paper/2402.00592/) (2024)
- [Rejection via Learning Density Ratios](https://mlsanity.com/paper/2405.18686/) (2024)
- [Consistent Learning-to-Defer with Expert-Conditional Advice](https://mlsanity.com/paper/2603.14324/) (2026)
- [Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction](https://mlsanity.com/paper/2603.18014/) (2026)
- [Conformalized Semi-supervised Random Forest for Classification and Abnormality Detection](https://mlsanity.com/paper/2302.02237/) (2023)

## Abstract (arXiv)

We investigate the problem of multiclass classification with rejection, where a classifier can choose not to make a prediction to avoid critical misclassification. First, we consider an approach based on simultaneous training of a classifier and a rejector, which achieves the state-of-the-art performance in the binary case. We analyze this approach for the multiclass case and derive a general condition for calibration to the Bayes-optimal solution, which suggests that calibration is hard to achieve by general loss functions unlike the binary case. Next, we consider another traditional approach based on confidence scores, in which the existing work focuses on a specific class of losses. We propose rejection criteria for more general losses for this approach and guarantee calibration to the Bayes-optimal solution. Finally, we conduct experiments to validate the relevance of our theoretical findings.