# A Statistical Test for Probabilistic Fairness

arXiv: https://arxiv.org/abs/2012.04800 · HTML: https://mlsanity.com/paper/2012.04800/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 25 of 25)

- [Statistical Analysis of Wasserstein Distributionally Robust Estimators](https://mlsanity.com/paper/2108.02120/) (2021)
- [Auditing Fairness by Betting](https://mlsanity.com/paper/2305.17570/) (2023)
- [Testing Group Fairness via Optimal Transport Projections](https://mlsanity.com/paper/2106.01070/) (2021)
- [Statistical Inference for Fairness Auditing](https://mlsanity.com/paper/2305.03712/) (2023)
- [Semi-Discrete Optimal Transport: Hardness, Regularization and Numerical Solution](https://mlsanity.com/paper/2103.06263/) (2021)
- [Optimal Transport of Classifiers to Fairness](https://mlsanity.com/paper/2202.03814/) (2022)
- [PROSAC: Provably Safe Certification for Machine Learning Models under Adversarial Attacks](https://mlsanity.com/paper/2402.02629/) (2024)
- [Wasserstein Robust Classification with Fairness Constraints](https://mlsanity.com/paper/2103.06828/) (2021)
- [Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools](https://mlsanity.com/paper/2309.17337/) (2023)
- [Robust ML Auditing using Prior Knowledge](https://mlsanity.com/paper/2505.04796/) (2025)
- [Explainable Global Fairness Verification of Tree-Based Classifiers](https://mlsanity.com/paper/2209.13179/) (2022)
- [Fairness and Sequential Decision Making: Limits, Lessons, and Opportunities](https://mlsanity.com/paper/2301.05753/) (2023)
- [Everything is Relative: Understanding Fairness with Optimal Transport](https://mlsanity.com/paper/2102.10349/) (2021)
- [Local Law 144: A Critical Analysis of Regression Metrics](https://mlsanity.com/paper/2302.04119/) (2023)
- [Specification Overfitting in Artificial Intelligence](https://mlsanity.com/paper/2403.08425/) (2024)
- [P2NIA: Privacy-Preserving Non-Iterative Auditing](https://mlsanity.com/paper/2504.00874/) (2025)
- [Towards clinical AI fairness: A translational perspective](https://mlsanity.com/paper/2304.13493/) (2023)
- [An Information-Flow Perspective on Algorithmic Fairness](https://mlsanity.com/paper/2312.10128/) (2023)
- [Disparate Conditional Prediction in Multiclass Classifiers](https://mlsanity.com/paper/2206.03234/) (2022)
- [A Critical Review of Predominant Bias in Neural Networks](https://mlsanity.com/paper/2502.11031/) (2025)
- [Testing for Causal Fairness](https://mlsanity.com/paper/2502.12874/) (2025)
- [From Fragile to Certified: Wasserstein Audits of Group Fairness Under Distribution Shift](https://mlsanity.com/paper/2509.26241/) (2025)
- [Cost Efficient Fairness Audit Under Partial Feedback](https://mlsanity.com/paper/2510.03734/) (2025)
- [Wasserstein projection distance for fairness testing of regression models](https://mlsanity.com/paper/2510.04114/) (2025)
- [Sequential Fairness Auditing with Limited Output Access](https://mlsanity.com/paper/2606.30338/) (2026)

## Abstract (arXiv)

Algorithms are now routinely used to make consequential decisions that affect human lives. Examples include college admissions, medical interventions or law enforcement. While algorithms empower us to harness all information hidden in vast amounts of data, they may inadvertently amplify existing biases in the available datasets. This concern has sparked increasing interest in fair machine learning, which aims to quantify and mitigate algorithmic discrimination. Indeed, machine learning models should undergo intensive tests to detect algorithmic biases before being deployed at scale. In this paper, we use ideas from the theory of optimal transport to propose a statistical hypothesis test for detecting unfair classifiers. Leveraging the geometry of the feature space, the test statistic quantifies the distance of the empirical distribution supported on the test samples to the manifold of distributions that render a pre-trained classifier fair. We develop a rigorous hypothesis testing mechanism for assessing the probabilistic fairness of any pre-trained logistic classifier, and we show both theoretically as well as empirically that the proposed test is asymptotically correct. In addition, the proposed framework offers interpretability by identifying the most favorable perturbation of the data so that the given classifier becomes fair.