# Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness

arXiv: https://arxiv.org/abs/2506.04193 · HTML: https://mlsanity.com/paper/2506.04193/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 22 of 22)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Reading Race: AI Recognises Patient's Racial Identity In Medical Images](https://mlsanity.com/paper/2107.10356/) (2021)
- [Identifying and Correcting Label Bias in Machine Learning](https://mlsanity.com/paper/1901.04966/) (2019)
- [Ethical Machine Learning in Health Care](https://mlsanity.com/paper/2009.10576/) (2020)
- [The Measure and Mismeasure of Fairness](https://mlsanity.com/paper/1808.00023/) (2018)
- [Problem Formulation and Fairness](https://mlsanity.com/paper/1901.02547/) (2019)
- [Double/Debiased/Neyman Machine Learning of Treatment Effects](https://mlsanity.com/paper/1701.08687/) (2017)
- [Causally motivated Shortcut Removal Using Auxiliary Labels](https://mlsanity.com/paper/2105.06422/) (2021)
- [An Empirical Characterization of Fair Machine Learning For Clinical Risk Prediction](https://mlsanity.com/paper/2007.10306/) (2020)
- [Diagnosing failures of fairness transfer across distribution shift in real-world medical settings](https://mlsanity.com/paper/2202.01034/) (2022)
- [Diagnosing Model Performance Under Distribution Shift](https://mlsanity.com/paper/2303.02011/) (2023)
- [Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making](https://mlsanity.com/paper/2302.06503/) (2023)
- [Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems](https://mlsanity.com/paper/2103.06172/) (2021)
- [Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare](https://mlsanity.com/paper/2202.01906/) (2022)
- [When accurate prediction models yield harmful self-fulfilling prophecies](https://mlsanity.com/paper/2312.01210/) (2023)
- [The Role of Subgroup Separability in Group-Fair Medical Image Classification](https://mlsanity.com/paper/2307.02791/) (2023)
- [Fairness and robustness in anti-causal prediction](https://mlsanity.com/paper/2209.09423/) (2022)

## Built on by (most cited first, 6 of 6)

- [Reflections from Research Roundtables at the Conference on Health, Inference, and Learning (CHIL) 2025](https://mlsanity.com/paper/2510.15217/) (2025)
- [Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations](https://mlsanity.com/paper/2510.24884/) (2025)
- [Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards](https://mlsanity.com/paper/2604.21769/) (2026)
- [meval: A Statistical Toolbox for Fine-Grained Model Performance Analysis](https://mlsanity.com/paper/2512.17409/) (2025)
- [Beyond Predictive Fairness: Quantifying Attribution Consistency Across Demographic Groups in Diabetic Retinopathy Screening](https://mlsanity.com/paper/2608.18759/) (2026)
- [Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation](https://mlsanity.com/paper/2609.20758/) (2026)

## Abstract (arXiv)

Disaggregated evaluation across subgroups is critical for assessing the fairness of machine learning models, but its uncritical use can mislead practitioners. We show that equal performance across subgroups is an unreliable measure of fairness when data are representative of the relevant populations but reflective of real-world disparities. Furthermore, when data are not representative due to selection bias, both disaggregated evaluation and alternative approaches based on conditional independence testing may be invalid without explicit assumptions regarding the bias mechanism. We use causal graphical models to characterize fairness properties and metric stability across subgroups under different data generating processes. Our framework suggests complementing disaggregated evaluations with explicit causal assumptions and analysis to control for confounding and distribution shift, including conditional independence testing and weighted performance estimation. These findings have broad implications for how practitioners design and interpret model assessments given the ubiquity of disaggregated evaluation.