# Neyman-Pearson Classification with Indecisions: Simultaneous Control of Selective Type I and Type II Errors

arXiv: https://arxiv.org/abs/2610.04233 · HTML: https://mlsanity.com/paper/2610.04233/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 3 Oct 2026.

## Builds on (most cited first, 19 of 19)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [Least Ambiguous Set-Valued Classifiers with Bounded Error Levels](https://mlsanity.com/paper/1609.00451/) (2016)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Selection by Prediction with Conformal p-values](https://mlsanity.com/paper/2210.01408/) (2022)
- [Neyman-Pearson (NP) classification algorithms and NP receiver operating characteristics (NP-ROC)](https://mlsanity.com/paper/1608.03109/) (2016)
- [Adaptive novelty detection with false discovery rate guarantee](https://mlsanity.com/paper/2208.06685/) (2022)
- [Selecting informative conformal prediction sets with false coverage rate control](https://mlsanity.com/paper/2403.12295/) (2024)
- [A Burden Shared is a Burden Halved: A Fairness-Adjusted Approach to Classification](https://mlsanity.com/paper/2110.05720/) (2021)
- [Neyman-Pearson Multi-class Classification via Cost-sensitive Learning](https://mlsanity.com/paper/2111.04597/) (2021)
- [A Joint Finite-Sample Certificate for Adaptive Selective Conformal Risk Control](https://mlsanity.com/paper/2606.08517/) (2026)
- [Asymmetric error control under imperfect supervision: a label-noise-adjusted Neyman-Pearson umbrella algorithm](https://mlsanity.com/paper/2112.00314/) (2021)
- [A Unified Framework for Large-Scale Inference of Classification: Error Rate Control and Optimality](https://mlsanity.com/paper/2504.07321/) (2025)
- [A Locally Adaptive Shrinkage Approach to False Selection Rate Control in High-Dimensional Classification](https://mlsanity.com/paper/2210.04268/) (2022)
- [Ask for More Than Bayes Optimal: A Theory of Indecisions for Selective Hypothesis Testing](https://mlsanity.com/paper/2412.12807/) (2024)

## Abstract (arXiv)

Binary classification often requires controlling both type I and type II error rates at prespecified levels. When the class distributions overlap substantially, the two error limits may be incompatible if every observation must be classified. We study Neyman-Pearson classification with indecisions (NPI), which permits uncertain observations to remain unclassified and assesses selective type I and type II errors among definitive classifications within each class. For an independently trained score, we develop a model-free calibration procedure that combines all candidate threshold pairs passing calibration into a rule whose decision region contains that of every passing candidate. We couple the entire threshold search to a common martingale within each class and compute its finite-horizon crossing probability exactly, avoiding a multiplicity adjustment for the number of candidate pairs. The resulting classifier satisfies both population selective error limits simultaneously with user-specified high probability in finite samples, without assumptions on the accuracy of the fitted score. We bound the additional indecision due to calibration and evaluate the procedure in simulations and applications to recidivism and credit-default prediction.