# Statistical Inference for Fairness Auditing

arXiv: https://arxiv.org/abs/2305.03712 · HTML: https://mlsanity.com/paper/2305.03712/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 5 May 2023.

## Builds on (most cited first, 17 of 17)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020)
- [Aequitas: A Bias and Fairness Audit Toolkit](https://mlsanity.com/paper/1811.05577/) (2018)
- [Auditing and Achieving Intersectional Fairness in Classification Problems](https://mlsanity.com/paper/1911.01468/) (2019)
- [Active Fairness Auditing](https://mlsanity.com/paper/2206.08450/) (2022)
- [A Statistical Test for Probabilistic Fairness](https://mlsanity.com/paper/2012.04800/) (2020)
- [Testing Group Fairness via Optimal Transport Projections](https://mlsanity.com/paper/2106.01070/) (2021)
- [Auditing ML Models for Individual Bias and Unfairness](https://mlsanity.com/paper/2003.05048/) (2020)
- [Conditional calibration for false discovery rate control under dependence](https://mlsanity.com/paper/2007.10438/) (2020)
- [Locating disparities in machine learning](https://mlsanity.com/paper/2208.06680/) (2022)
- [Fairness Uncertainty Quantification: How certain are you that the model is fair?](https://mlsanity.com/paper/2304.13950/) (2023)

## Built on by (most cited first, 20 of 20)

- [Rethinking Explainable Machine Learning as Applied Statistics](https://mlsanity.com/paper/2402.02870/) (2024)
- [Conformalizing Machine Translation Evaluation](https://mlsanity.com/paper/2306.06221/) (2023)
- [Fairness Hacking: The Malicious Practice of Shrouding Unfairness in Algorithms](https://mlsanity.com/paper/2311.06826/) (2023)
- [The Intersectionality Problem for Algorithmic Fairness](https://mlsanity.com/paper/2411.02569/) (2024)
- [Exposing the Illusion of Fairness: Auditing Vulnerabilities to Distributional Manipulation Attacks](https://mlsanity.com/paper/2507.20708/) (2025)
- [From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms](https://mlsanity.com/paper/2502.08166/) (2025)
- [On the Origins of Sampling Bias: Implications on Fairness Measurement and Mitigation](https://mlsanity.com/paper/2503.17956/) (2025)
- [Size-adaptive Hypothesis Testing for Fairness](https://mlsanity.com/paper/2506.10586/) (2025)
- [Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models](https://mlsanity.com/paper/2606.09046/) (2026)
- [Peer-induced Fairness: A Causal Approach for Algorithmic Fairness Auditing](https://mlsanity.com/paper/2408.02558/) (2024)
- [A Critical Review of Predominant Bias in Neural Networks](https://mlsanity.com/paper/2502.11031/) (2025)
- [The Fair Game: Auditing & Debiasing AI Algorithms Over Time](https://mlsanity.com/paper/2508.06443/) (2025)
- [Cost Efficient Fairness Audit Under Partial Feedback](https://mlsanity.com/paper/2510.03734/) (2025)
- [An Explainable and Fair AI Tool for PCOS Risk Assessment: Calibration, Subgroup Equity, and Interactive Clinical Deployment](https://mlsanity.com/paper/2511.11636/) (2025)
- [meval: A Statistical Toolbox for Fine-Grained Model Performance Analysis](https://mlsanity.com/paper/2512.17409/) (2025)
- [Investigating Data Interventions for Subgroup Fairness: An ICU Case Study](https://mlsanity.com/paper/2604.03478/) (2026)
- [Fairness in multi-class multi-group classification problems via contextial coherent risk measures](https://mlsanity.com/paper/2608.30223/) (2026)
- [Selection-Aware Stress Testing for Interactive Agents](https://mlsanity.com/paper/2608.30916/) (2026)
- [Tolerance-Based Fairness Auditing: Violation Certification and Sensitivity Screening](https://mlsanity.com/paper/2610.01005/) (2026)
- [Testing Conditional Distribution Equality Using Generative Models](https://mlsanity.com/paper/2610.20447/) (2026)

## Abstract (arXiv)

Before deploying a black-box model in high-stakes problems, it is important to evaluate the model's performance on sensitive subpopulations. For example, in a recidivism prediction task, we may wish to identify demographic groups for which our prediction model has unacceptably high false positive rates or certify that no such groups exist. In this paper, we frame this task, often referred to as "fairness auditing," in terms of multiple hypothesis testing. We show how the bootstrap can be used to simultaneously bound performance disparities over a collection of groups with statistical guarantees. Our methods can be used to flag subpopulations affected by model underperformance, and certify subpopulations for which the model performs adequately. Crucially, our audit is model-agnostic and applicable to nearly any performance metric or group fairness criterion. Our methods also accommodate extremely rich -- even infinite -- collections of subpopulations. Further, we generalize beyond subpopulations by showing how to assess performance over certain distribution shifts. We test the proposed methods on benchmark datasets in predictive inference and algorithmic fairness and find that our audits can provide interpretable and trustworthy guarantees.