# ABROCA Distributions For Algorithmic Bias Assessment: Considerations Around Interpretation

arXiv: https://arxiv.org/abs/2411.19090 · HTML: https://mlsanity.com/paper/2411.19090/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 17 references that are in the map.
- First posted on arXiv 28 Nov 2024.

## Builds on (most cited first, 3 of 3)

- [The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons](https://mlsanity.com/paper/1912.04930/) (2019)
- [Towards Equity and Algorithmic Fairness in Student Grade Prediction](https://mlsanity.com/paper/2105.06604/) (2021)
- [Is Your Model "MADD"? A Novel Metric to Evaluate Algorithmic Fairness for Predictive Student Models](https://mlsanity.com/paper/2305.15342/) (2023)

## Built on by (most cited first, 4 of 4)

- [Toward Sufficient Statistical Power in Algorithmic Bias Assessment: A Test for ABROCA](https://mlsanity.com/paper/2501.04683/) (2025)
- [Achieving Fair Skin Lesion Detection through Skin Tone Normalization and Channel Pruning](https://mlsanity.com/paper/2509.22712/) (2025)
- [FRAME: Separating sampling variation from performance disparities in medical image analysis](https://mlsanity.com/paper/2608.25981/) (2026)
- [Accuracy Is Not Enough: A Cross-Architecture Audit of Demographic Bias in Deep Knowledge Tracing](https://mlsanity.com/paper/2609.20249/) (2026)

## Abstract (arXiv)

Algorithmic bias continues to be a key concern of learning analytics. We study the statistical properties of the Absolute Between-ROC Area (ABROCA) metric. This fairness measure quantifies group-level differences in classifier performance through the absolute difference in ROC curves. ABROCA is particularly useful for detecting nuanced performance differences even when overall Area Under the ROC Curve (AUC) values are similar. We sample ABROCA under various conditions, including varying AUC differences and class distributions. We find that ABROCA distributions exhibit high skewness dependent on sample sizes, AUC differences, and class imbalance. When assessing whether a classifier is biased, this skewness inflates ABROCA values by chance, even when data is drawn (by simulation) from populations with equivalent ROC curves. These findings suggest that ABROCA requires careful interpretation given its distributional properties, especially when used to assess the degree of bias and when classes are imbalanced.