# Limitations of Pinned AUC for Measuring Unintended Bias

arXiv: https://arxiv.org/abs/1903.02088 · HTML: https://mlsanity.com/paper/1903.02088/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2019.

## Built on by (most cited first, 4 of 4)

- [Handling Bias in Toxic Speech Detection: A Survey](https://mlsanity.com/paper/2202.00126/) (2022)
- [Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs](https://mlsanity.com/paper/2601.03087/) (2026)
- [Determination of toxic comments and unintended model bias minimization using Deep learning approach](https://mlsanity.com/paper/2311.04789/) (2023)
- [Look Inside Each Video: Rethinking How Video Anomaly Detection Is Evaluated](https://mlsanity.com/paper/2608.21854/) (2026)

## Abstract (arXiv)

This report examines the Pinned AUC metric introduced and highlights some of its limitations. Pinned AUC provides a threshold-agnostic measure of unintended bias in a classification model, inspired by the ROC-AUC metric. However, as we highlight in this report, there are ways that the metric can obscure different kinds of unintended biases when the underlying class distributions on which bias is being measured are not carefully controlled.