# Toward Sufficient Statistical Power in Algorithmic Bias Assessment: A Test for ABROCA

arXiv: https://arxiv.org/abs/2501.04683 · HTML: https://mlsanity.com/paper/2501.04683/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 26 references that are in the map.
- First posted on arXiv 8 Jan 2025.

## Builds on (most cited first, 8 of 8)

- [The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons](https://mlsanity.com/paper/1912.04930/) (2019)
- [Algorithmic Fairness in Education](https://mlsanity.com/paper/2007.05443/) (2020)
- [Towards Equity and Algorithmic Fairness in Student Grade Prediction](https://mlsanity.com/paper/2105.06604/) (2021)
- [Is Your Model "MADD"? A Novel Metric to Evaluate Algorithmic Fairness for Predictive Student Models](https://mlsanity.com/paper/2305.15342/) (2023)
- [ABROCA Distributions For Algorithmic Bias Assessment: Considerations Around Interpretation](https://mlsanity.com/paper/2411.19090/) (2024)
- [Protected Attributes Tell Us Who, Behavior Tells Us How: A Comparison of Demographic and Behavioral Oversampling for Fair Student Success Modeling](https://mlsanity.com/paper/2212.10166/) (2022)
- [Predicting Early Dropout: Calibration and Algorithmic Fairness Considerations](https://mlsanity.com/paper/2103.09068/) (2021)
- [Evaluating Algorithmic Bias in Models for Predicting Academic Performance of Filipino Students](https://mlsanity.com/paper/2405.09821/) (2024)

## Built on by (most cited first, 1 of 1)

- [Toward Measuring AI's Effects on Skill Formation: The Stock-Formation Gap](https://mlsanity.com/paper/2605.16283/) (2026)

## Abstract (arXiv)

Algorithmic bias is a pressing concern in educational data mining (EDM), as it risks amplifying inequities in learning outcomes. The Area Between ROC Curves (ABROCA) metric is frequently used to measure discrepancies in model performance across demographic groups to quantify overall model fairness. However, its skewed distribution--especially when class or group imbalances exist--makes significance testing challenging. This study investigates ABROCA's distributional properties and contributes robust methods for its significance testing. Specifically, we address (1) whether ABROCA follows any known distribution, (2) how to reliably test for algorithmic bias using ABROCA, and (3) the statistical power achievable with ABROCA-based bias assessments under typical EDM sample specifications. Simulation results confirm that ABROCA does not match standard distributions, including those suited to accommodate skewness. We propose nonparametric randomization tests for ABROCA and demonstrate that reliably detecting bias with ABROCA requires large sample sizes or substantial effect sizes, particularly in imbalanced settings. Findings suggest that ABROCA-based bias evaluations based on sample sizes common in EDM tend to be underpowered, undermining the reliability of conclusions about model fairness. By offering open-source code to simulate power and statistically test ABROCA, this paper aims to foster more reliable statistical testing in EDM research. It supports broader efforts toward replicability and equity in educational modeling.