# Identifying Significant Predictive Bias in Classifiers

arXiv: https://arxiv.org/abs/1611.08292 · HTML: https://mlsanity.com/paper/1611.08292/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2016.

## Built on by (most cited first, 15 of 15)

- [Trustworthy AI: A Computational Perspective](https://mlsanity.com/paper/2107.06641/) (2021)
- [Locating disparities in machine learning](https://mlsanity.com/paper/2208.06680/) (2022)
- [Automated Supervised Feature Selection for Differentiated Patterns of Care](https://mlsanity.com/paper/2111.03495/) (2021)
- [Bias Detection via Maximum Subgroup Discrepancy](https://mlsanity.com/paper/2502.02221/) (2025)
- [Post-discovery Analysis of Anomalous Subsets](https://mlsanity.com/paper/2111.14622/) (2021)
- [Provable Detection of Propagating Sampling Bias in Prediction Models](https://mlsanity.com/paper/2302.06752/) (2023)
- [Feature Importance Disparities for Data Bias Investigations](https://mlsanity.com/paper/2303.01704/) (2023)
- [Interpretable Differencing of Machine Learning Models](https://mlsanity.com/paper/2306.06473/) (2023)
- [Non-Comparative Fairness for Human-Auditing and Its Relation to Traditional Fairness Notions](https://mlsanity.com/paper/2107.01277/) (2021)
- [An Efficient Framework for Monitoring Subgroup Performance of Machine Learning Systems](https://mlsanity.com/paper/2212.08312/) (2022)
- [Insufficiently Justified Disparate Impact: A New Criterion for Subgroup Fairness](https://mlsanity.com/paper/2306.11181/) (2023)
- [Auditing Predictive Models for Intersectional Biases](https://mlsanity.com/paper/2306.13064/) (2023)
- [Explaining Knock-on Effects of Bias Mitigation](https://mlsanity.com/paper/2312.00765/) (2023)
- [SubROC: AUC-Based Discovery of Exceptional Subgroup Performance for Binary Classifiers](https://mlsanity.com/paper/2505.11283/) (2025)
- [meval: A Statistical Toolbox for Fine-Grained Model Performance Analysis](https://mlsanity.com/paper/2512.17409/) (2025)

## Abstract (arXiv)

We present a novel subset scan method to detect if a probabilistic binary classifier has statistically significant bias -- over or under predicting the risk -- for some subgroup, and identify the characteristics of this subgroup. This form of model checking and goodness-of-fit test provides a way to interpretably detect the presence of classifier bias or regions of poor classifier fit. This allows consideration of not just subgroups of a priori interest or small dimensions, but the space of all possible subgroups of features. To address the difficulty of considering these exponentially many possible subgroups, we use subset scan and parametric bootstrap-based methods. Extending this method, we can penalize the complexity of the detected subgroup and also identify subgroups with high classification errors. We demonstrate these methods and find interesting results on the COMPAS crime recidivism and credit delinquency data.