# Counterfactually Comparing Abstaining Classifiers

arXiv: https://arxiv.org/abs/2305.10564 · HTML: https://mlsanity.com/paper/2305.10564/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 73 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 20 of 20)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Time-uniform, nonparametric, nonasymptotic confidence sequences](https://mlsanity.com/paper/1810.08240/) (2018)
- [Game-theoretic statistics and safe anytime-valid inference](https://mlsanity.com/paper/2210.01948/) (2022)
- [Semiparametric doubly robust targeted double machine learning: a review](https://mlsanity.com/paper/2203.06469/) (2022)
- [Machine Learning with a Reject Option: A survey](https://mlsanity.com/paper/2107.11277/) (2021)
- [The Algorithmic Automation Problem: Prediction, Triage, and Human Effort](https://mlsanity.com/paper/1903.12220/) (2019)
- [Safe Testing](https://mlsanity.com/paper/1906.07801/) (2019)
- [Selective Classification via One-Sided Prediction](https://mlsanity.com/paper/2010.07853/) (2020)
- [Nonparametric causal effects based on incremental propensity score interventions](https://mlsanity.com/paper/1704.00211/) (2017)
- [Classification with abstention but without disparities](https://mlsanity.com/paper/2102.12258/) (2021)
- [Visually Communicating and Teaching Intuition for Influence Functions](https://mlsanity.com/paper/1810.03260/) (2018)
- [On Fairness, Diversity and Randomness in Algorithmic Decision Making](https://mlsanity.com/paper/1706.10208/) (2017)
- [Performance measures for classification systems with rejection](https://mlsanity.com/paper/1504.02763/) (2015)
- [Causal Inference: A Missing Data Perspective](https://mlsanity.com/paper/1712.06170/) (2017)
- [Sensitivity analysis via the proportion of unmeasured confounding](https://mlsanity.com/paper/1912.02793/) (2019)
- [On Adaptive Propensity Score Truncation in Causal Inference](https://mlsanity.com/paper/1707.05861/) (2017)

## Built on by (most cited first, 2 of 2)

- [A Causal Framework for Evaluating Deferring Systems](https://mlsanity.com/paper/2405.18902/) (2024)
- [VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection](https://mlsanity.com/paper/2607.06254/) (2026)

## Abstract (arXiv)

Abstaining classifiers have the option to abstain from making predictions on inputs that they are unsure about. These classifiers are becoming increasingly popular in high-stakes decision-making problems, as they can withhold uncertain predictions to improve their reliability and safety. When evaluating black-box abstaining classifier(s), however, we lack a principled approach that accounts for what the classifier would have predicted on its abstentions. These missing predictions matter when they can eventually be utilized, either directly or as a backup option in a failure mode. In this paper, we introduce a novel approach and perspective to the problem of evaluating and comparing abstaining classifiers by treating abstentions as missing data. Our evaluation approach is centered around defining the counterfactual score of an abstaining classifier, defined as the expected performance of the classifier had it not been allowed to abstain. We specify the conditions under which the counterfactual score is identifiable: if the abstentions are stochastic, and if the evaluation data is independent of the training data (ensuring that the predictions are missing at random), then the score is identifiable. Note that, if abstentions are deterministic, then the score is unidentifiable because the classifier can perform arbitrarily poorly on its abstentions. Leveraging tools from observational causal inference, we then develop nonparametric and doubly robust methods to efficiently estimate this quantity under identification. Our approach is examined in both simulated and real data experiments.