# Size-adaptive Hypothesis Testing for Fairness

arXiv: https://arxiv.org/abs/2506.10586 · HTML: https://mlsanity.com/paper/2506.10586/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 27 of 27)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016)
- [Multiaccuracy: Black-Box Post-Processing for Fairness in Classification](https://mlsanity.com/paper/1805.12317/) (2018)
- [A comparative study of fairness-enhancing interventions in machine learning](https://mlsanity.com/paper/1802.04422/) (2018)
- [Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty](https://mlsanity.com/paper/2011.07586/) (2020)
- [An Intersectional Definition of Fairness](https://mlsanity.com/paper/1807.08362/) (2018)
- [Fairlearn: Assessing and Improving Fairness of AI Systems](https://mlsanity.com/paper/2303.16626/) (2023)
- [Characterizing Intersectional Group Fairness with Worst-Case Comparisons](https://mlsanity.com/paper/2101.01673/) (2021)
- [A Survey on Intersectional Fairness in Machine Learning: Notions, Mitigation, and Challenges](https://mlsanity.com/paper/2305.06969/) (2023)
- [A survey of bias in Machine Learning through the prism of Statistical Parity for the Adult Data Set](https://mlsanity.com/paper/2003.14263/) (2020)
- [Auditing and Achieving Intersectional Fairness in Classification Problems](https://mlsanity.com/paper/1911.01468/) (2019)
- [Bayesian Modeling of Intersectional Fairness: The Variance of Bias](https://mlsanity.com/paper/1811.07255/) (2018)
- [FairLens: Auditing Black-box Clinical Decision Support Systems](https://mlsanity.com/paper/2011.04049/) (2020)
- [Statistical Inference for Fairness Auditing](https://mlsanity.com/paper/2305.03712/) (2023)
- [Enforcing Group Fairness in Algorithmic Decision Making: Utility Maximization Under Sufficiency](https://mlsanity.com/paper/2206.02237/) (2022)
- [Maxmin-Fair Ranking: Individual Fairness under Group-Fairness Constraints](https://mlsanity.com/paper/2106.08652/) (2021)
- [Uncertainty-based Fairness Measures](https://mlsanity.com/paper/2312.11299/) (2023)
- [Fairness Score and Process Standardization: Framework for Fairness Certification in Artificial Intelligence Systems](https://mlsanity.com/paper/2201.06952/) (2022)
- [Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds](https://mlsanity.com/paper/2004.12332/) (2020)
- [Bounding and Approximating Intersectional Fairness through Marginal Fairness](https://mlsanity.com/paper/2206.05828/) (2022)
- [Confidence Intervals for Testing Disparate Impact in Fair Learning](https://mlsanity.com/paper/1807.06362/) (2018)
- [The Intersectionality Problem for Algorithmic Fairness](https://mlsanity.com/paper/2411.02569/) (2024)

## Built on by (most cited first, 1 of 1)

- [Operationalizing Fairness in Text-to-Image Models: A Survey of Bias, Fairness Audits and Mitigation Strategies](https://mlsanity.com/paper/2604.16516/) (2026)

## Abstract (arXiv)

Determining whether an algorithmic decision-making system discriminates against a specific demographic typically involves comparing a single point estimate of a fairness metric against a predefined threshold. This practice is statistically brittle: it ignores sampling error and treats small demographic subgroups the same as large ones. The problem intensifies in intersectional analyses, where multiple sensitive attributes are considered jointly, giving rise to a larger number of smaller groups. As these groups become more granular, the data representing them becomes too sparse for reliable estimation, and fairness metrics yield excessively wide confidence intervals, precluding meaningful conclusions about potential unfair treatments. In this paper, we introduce a unified, size-adaptive, hypothesis-testing framework that turns fairness assessment into an evidence-based statistical decision. Our contribution is twofold. (i) For sufficiently large subgroups, we prove a Central-Limit result for the statistical parity difference, leading to analytic confidence intervals and a Wald test whose type-I (false positive) error is guaranteed at level α. (ii) For the long tail of small intersectional groups, we derive a fully Bayesian Dirichlet-multinomial estimator; Monte-Carlo credible intervals are calibrated for any sample size and naturally converge to Wald intervals as more data becomes available. We validate our approach empirically on benchmark datasets, demonstrating how our tests provide interpretable, statistically rigorous decisions under varying degrees of data availability and intersectionality.