# Auditing ML Models for Individual Bias and Unfairness

arXiv: https://arxiv.org/abs/2003.05048 · HTML: https://mlsanity.com/paper/2003.05048/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2020.

## Built on by (most cited first, 18 of 18)

- [Testing Group Fairness via Optimal Transport Projections](https://mlsanity.com/paper/2106.01070/) (2021)
- [Statistical Inference for Fairness Auditing](https://mlsanity.com/paper/2305.03712/) (2023)
- [From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing](https://mlsanity.com/paper/2410.04772/) (2024)
- [Statistical inference for individual fairness](https://mlsanity.com/paper/2103.16714/) (2021)
- [PROSAC: Provably Safe Certification for Machine Learning Models under Adversarial Attacks](https://mlsanity.com/paper/2402.02629/) (2024)
- [An External Stability Audit Framework to Test the Validity of Personality Prediction in AI Hiring](https://mlsanity.com/paper/2201.09151/) (2022)
- [Racial Bias within Face Recognition: A Survey](https://mlsanity.com/paper/2305.00817/) (2023)
- [TeleSparse: Practical Privacy-Preserving Verification of Deep Neural Networks](https://mlsanity.com/paper/2504.19274/) (2025)
- [Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairness](https://mlsanity.com/paper/2303.00673/) (2023)
- [Is this model reliable for everyone? Testing for strong calibration](https://mlsanity.com/paper/2307.15247/) (2023)
- [Everything is Relative: Understanding Fairness with Optimal Transport](https://mlsanity.com/paper/2102.10349/) (2021)
- [Fair Representation Learning using Interpolation Enabled Disentanglement](https://mlsanity.com/paper/2108.00295/) (2021)
- [Fair mapping](https://mlsanity.com/paper/2209.00617/) (2022)
- [Peer-induced Fairness: A Causal Approach for Algorithmic Fairness Auditing](https://mlsanity.com/paper/2408.02558/) (2024)
- [From Fragile to Certified: Wasserstein Audits of Group Fairness Under Distribution Shift](https://mlsanity.com/paper/2509.26241/) (2025)
- [Cost Efficient Fairness Audit Under Partial Feedback](https://mlsanity.com/paper/2510.03734/) (2025)
- [Wasserstein projection distance for fairness testing of regression models](https://mlsanity.com/paper/2510.04114/) (2025)
- [A Geometric Theory of Robust Fairness Audits](https://mlsanity.com/paper/2608.24818/) (2026)

## Abstract (arXiv)

We consider the task of auditing ML models for individual bias/unfairness. We formalize the task in an optimization problem and develop a suite of inferential tools for the optimal value. Our tools permit us to obtain asymptotic confidence intervals and hypothesis tests that cover the target/control the Type I error rate exactly. To demonstrate the utility of our tools, we use them to reveal the gender and racial biases in Northpointe's COMPAS recidivism prediction instrument.