# Neyman-Pearson (NP) classification algorithms and NP receiver operating characteristics (NP-ROC)

arXiv: https://arxiv.org/abs/1608.03109 · HTML: https://mlsanity.com/paper/1608.03109/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2016.

## Built on by (most cited first, 23 of 23)

- [Meta-Cal: Well-controlled Post-hoc Calibration by Ranking](https://mlsanity.com/paper/2105.04290/) (2021)
- [Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers](https://mlsanity.com/paper/2208.11111/) (2022)
- [FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees](https://mlsanity.com/paper/2411.02603/) (2024)
- [Neyman-Pearson Multi-class Classification via Cost-sensitive Learning](https://mlsanity.com/paper/2111.04597/) (2021)
- [FaiREE: Fair Classification with Finite-Sample and Distribution-Free Guarantee](https://mlsanity.com/paper/2211.15072/) (2022)
- [Federated Learning with Convex Global and Local Constraints](https://mlsanity.com/paper/2310.10117/) (2023)
- [What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents](https://mlsanity.com/paper/2607.22868/) (2026)
- [Asymmetric error control under imperfect supervision: a label-noise-adjusted Neyman-Pearson umbrella algorithm](https://mlsanity.com/paper/2112.00314/) (2021)
- [Distribution-Free Rates in Neyman-Pearson Classification](https://mlsanity.com/paper/2402.09560/) (2024)
- [Meta-Statistical Learning: Supervised Learning of Statistical Estimators](https://mlsanity.com/paper/2502.12088/) (2025)
- [Density Ratio Estimation and Neyman Pearson Classification with Missing Data](https://mlsanity.com/paper/2302.10655/) (2023)
- [GBM-based Bregman Proximal Algorithms for Constrained Learning](https://mlsanity.com/paper/2308.10767/) (2023)
- [Learning Pareto-Efficient Decisions with Confidence](https://mlsanity.com/paper/2110.09864/) (2021)
- [Hierarchical Neyman-Pearson Classification for Prioritizing Severe Disease Categories in COVID-19 Patient Data](https://mlsanity.com/paper/2210.02197/) (2022)
- [Ensemble Threshold Calibration for Stable Sensitivity Control](https://mlsanity.com/paper/2510.02116/) (2025)
- [PatchOptic for Shared-State LLM Workflows with Projected Views and Verified Structured Updates](https://mlsanity.com/paper/2607.05483/) (2026)
- [Primal--Dual Alternating Neural Learning for Timely Classification with Performance Guarantees](https://mlsanity.com/paper/2608.23480/) (2026)
- [Classification with Abstention Under Class-Conditional Error Constraints](https://mlsanity.com/paper/2609.22632/) (2026)
- [NEEDLEWORK: Offline Rewriting of Robot Data with Verified Local Stitches](https://mlsanity.com/paper/2610.02339/) (2026)
- [Neyman-Pearson Classification with Indecisions: Simultaneous Control of Selective Type I and Type II Errors](https://mlsanity.com/paper/2610.04233/) (2026)
- [From Probe Scores to Alarm Policies: Operational Validity of Activation Monitors for Language-Model Agents](https://mlsanity.com/paper/2610.04575/) (2026)
- [Accuracy and Uncertainty in Machine Deception Detection (Research White Paper)](https://mlsanity.com/paper/2610.15314/) (2026)
- [One-Sided Verification Under Limited Expert Review](https://mlsanity.com/paper/2610.18089/) (2026)

## Abstract (arXiv)

In many binary classification applications such as disease diagnosis and spam detection, practitioners often face great needs to control type I errors (i.e., the conditional probability of misclassifying a class 0 observation as class 1) so that it remains below a desired threshold. To address this need, the Neyman-Pearson (NP) classification paradigm is a natural choice; it minimizes type II error (i.e., the conditional probability of misclassifying a class 1 observation as class 0) while enforcing an upper bound, α, on the type I error. Although the NP paradigm has a century-long history in hypothesis testing, it has not been well recognized and implemented in classification schemes. Common practices that directly limit the empirical type I error to no more than α do not satisfy the type I error control objective because the resulting classifiers are still likely to have type I errors much larger than α. As a result, the NP paradigm has not been properly implemented for many classification scenarios in practice. In this work, we develop the first umbrella algorithm that implements the NP paradigm for all scoring-type classification methods, including popular methods such as logistic regression, support vector machines and random forests. Powered by this umbrella algorithm, we propose a novel graphical tool for NP classification methods: NP receiver operating characteristic (NP-ROC) bands, motivated by the popular receiver operating characteristic (ROC) curves. NP-ROC bands will help choose α in a data adaptive way and compare different NP classifiers. We demonstrate the use and properties of the NP umbrella algorithm and NP-ROC bands, available in the R package nproc, through simulation and real data case studies.