mlsanityData to 7 Oct 2026

Machine Learning with a Reject Option: A survey

arXiv 2107.11277 ↗First posted 23 Jul 2021cs.LG cs.AIcited by 107

Not on the map. Why?

Abstract (arXiv)

Machine learning models always make a prediction, even when it is likely to be inaccurate. This behavior should be avoided in many decision support applications, where mistakes can have severe consequences. Albeit already studied in 1970, machine learning with rejection recently gained interest. This machine learning subfield enables machine learning models to abstain from making a prediction when likely to make a mistake. This survey aims to provide an overview on machine learning with rejection. We introduce the conditions leading to two types of rejection, ambiguity and novelty rejection, which we carefully formalize. Moreover, we review and categorize strategies to evaluate a model's predictive and rejective quality. Additionally, we define the existing architectures for models with rejection and describe the standard techniques for learning such models. Finally, we provide examples of relevant application domains and show how machine learning with rejection relates to other machine learning research areas.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 45 of its 275 references that are in the map. First posted on arXiv 23 Jul 2021.

Builds on

45 of 275 references.

References · 45

  1. 2017Selective Classification for Deep Neural Networkscited by 799
  2. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  3. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
  4. 2020Consistent Estimators for Learning to Defer to an Expertcited by 262
  5. 2018Meta-Learning: A Surveycited by 254
  6. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 228
  7. 2017Concrete Dropoutcited by 227
  8. 2018To Trust Or Not To Trust A Classifiercited by 205
  9. 2020Self-Adaptive Training: beyond Empirical Risk Minimizationcited by 126
  10. 2019Combating Label Noise in Deep Learning Using Abstentioncited by 86
  11. 2019Deep Gamblers: Learning to Abstain with Portfolio Theorycited by 69
  12. 2021Optimal strategies for reject option classifierscited by 61
33 more
  1. 2020Classification with Rejection Based on Cost-sensitive Classificationcited by 55
  2. 2020Classification Under Human Assistancecited by 42
  3. 2019On the Calibration of Multiclass Classification with Rejectioncited by 41
  4. 2019Hybrid Models with Deep and Invertible Featurescited by 41
  5. 2020Selective Classification via One-Sided Predictioncited by 37
  6. 2020Confidence Estimation via Auxiliary Modelscited by 33
  7. 2019Deep Neural Rejection against Adversarial Examplescited by 31
  8. 2017Online Learning with Abstentioncited by 28
  9. 2020Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detectioncited by 26
  10. 2018Uncertainty-Aware Attention for Reliable Interpretation and Predictioncited by 24
  11. 2019Epistemic Uncertainty Samplingcited by 24
  12. 2021Exponential Savings in Agnostic Active Learning through Abstentioncited by 23
  13. 2014Beyond Disagreement-based Agnostic Active Learningcited by 21
  14. 2017Blocking Transferability of Adversarial Examples in Black-Box Learning Systemscited by 19
  15. 2020Asymmetric Distribution Measure for Few-shot Learningcited by 18
  16. 2015Performance measures for classification systems with rejectioncited by 16
  17. 2021Two Coupled Rejection Metrics Can Tell Adversarial Examples Apartcited by 14
  18. 2020Versatile Verification of Tree Ensemblescited by 12
  19. 2020Fast and Accurate k-means++ via Rejection Samplingcited by 10
  20. 2013Classifiers With a Reject Option for Early Time-Series Classificationcited by 9
  21. 2017On Reject and Refine Options in Multicategory Classificationcited by 9
  22. 2021SCRIB: Set-classifier with Class-specific Risk Bounds for Blackbox Modelscited by 9
  23. 2020ATRO: Adversarial Training with a Rejection Optioncited by 8
  24. 2019Reliable Multi-label Classification: Prediction with Partial Abstentioncited by 6
  25. 2017SafePredict: A Meta-Algorithm for Machine Learning That Uses Refusals to Guarantee Correctnesscited by 5
  26. 2018Machine Learning with Abstention for Automated Liver Disease Diagnosiscited by 5
  27. 2019Binary Classification with Bounded Abstention Ratecited by 3
  28. 2015Optimum Reject Options for Prototype-based Classificationcited by 2
  29. 2018COBRAS-TS: A new approach to Semi-Supervised Clustering of Time Seriescited by 2
  30. 2020A Novel Meta Learning Framework for Feature Selection using Data Synthesis and Fuzzy Similaritycited by 2
  31. 2015Robust hyperspectral image classification with rejection fieldscited by 1
  32. 2015Image Classification with Rejection using Contextual Informationcited by 1
  33. 2016Learning to Abstain from Binary Predictioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 11; 2023: 21; 2024: 28; 2025: 25; 2026: 21.

107 papers.

Sort:

By area:

  1. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
  2. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
  3. 2023Explaining Predictive Uncertainty with Information Theoretic Shapley Valuescited by 36
  4. 2023Learning to Reject with a Fixed Predictor: Application to Decontextualizationcited by 32
  5. 2022Learning When to Advise Human Decision Makerscited by 18
  6. 2023How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networkscited by 15
  7. 2023Plugin estimators for selective classification with out-of-distribution detectioncited by 14
  8. 2023Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibrationcited by 14
  9. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  10. 2022AUC-based Selective Classificationcited by 13
  11. 2024A Causal Framework for Evaluating Deferring Systemscited by 13
  12. 2024A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimatorscited by 13
38 more
  1. 2024A2C: A Modular Multi-stage Collaborative Decision Framework for Human-AI Teamscited by 12
  2. 2022"Even if ..." -- Diverse Semifactual Explanations of Rejectcited by 11
  3. 2023A Review of and Roadmap for Data Science and Machine Learning for the Neuropsychiatric Phenotype of Autismcited by 11
  4. 2025Selective Conformal Risk Controlcited by 10
  5. 2023Unsupervised Anomaly Detection with Rejectioncited by 9
  6. 2022A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunitiescited by 8
  7. 2024Machine Learning Robustness: A Primercited by 8
  8. 2022Model Agnostic Local Explanations of Rejectcited by 7
  9. 2024AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systemscited by 7
  10. 2025PredictaBoard: Benchmarking LLM Score Predictabilitycited by 7
  11. 2026Agentic Business Process Management: A Research Manifestocited by 7
  12. 2023Survey on Leveraging Uncertainty Estimation Towards Trustworthy Deep Neural Networks: The Case of Reject Option and Post-training Processingcited by 6
  13. 2025PAC Learning with Improvementscited by 6
  14. 2026Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Futurecited by 6
  15. 2022Bayesian autoencoders with uncertainty quantification: Towards trustworthy anomaly detectioncited by 4
  16. 2023Empirical Optimal Risk to Quantify Model Trustworthiness for Failure Detectioncited by 4
  17. 2023Online Decision Mediationcited by 4
  18. 2024Fast and reliable uncertainty quantification with neural network ensembles for industrial image classificationcited by 4
  19. 2024Classification with Conceptual Safeguardscited by 4
  20. 2025How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyondcited by 4
  21. 2022Evaluating Generalizability of Fine-Tuned Models for Fake News Detectioncited by 3
  22. 2023Fast Online Value-Maximizing Prediction Sets with Conformal Cost Controlcited by 3
  23. 2023Finding Competence Regions in Domain Generalizationcited by 3
  24. 2024Bandits with Abstention under Expert Advicecited by 3
  25. 2024Rejection via Learning Density Ratioscited by 3
  26. 2024Safety Monitoring of Machine Learning Perception Functions: a Surveycited by 3
  27. 2025Classification with Reject Option: Distribution-free Error Guarantees via Conformal Predictioncited by 3
  28. 2022A Case for Rejection in Low Resource ML Deploymentcited by 2
  29. 2023Counterfactually Comparing Abstaining Classifierscited by 2
  30. 2024Near OOD Detection for Vision-Language Prompt Learning with Contrastive Logit Scorecited by 2
  31. 2024Finding Patterns in Ambiguity: Interpretable Stress Testing in the Decision~Boundarycited by 2
  32. 2024Learning To Help: Training Models to Assist Legacy Devicescited by 2
  33. 2024On the Robustness of Adversarial Training Against Uncertainty Attackscited by 2
  34. 2025Interpretable and Fair Mechanisms for Abstaining Classifierscited by 2
  35. 2025Knowing What You Cannot Explain: Learning to Reject Low-Quality Explanationscited by 2
  36. 2025Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensemblescited by 2
  37. 2025Catch Your Breath: Adaptive Computation for Self-Paced Sequence Productioncited by 2
  38. 2025SHRUG-FM: Reliability-Aware Foundation Models for Earth Observationcited by 2

50 most cited of 107 All 107 (JSON)