# Multi-Group Fairness Evaluation via Conditional Value-at-Risk Testing

arXiv: https://arxiv.org/abs/2312.03867 · HTML: https://mlsanity.com/paper/2312.03867/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Builds on (most cited first, 16 of 16)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [On Fairness and Calibration](https://mlsanity.com/paper/1709.02012/) (2017)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [An Intersectional Definition of Fairness](https://mlsanity.com/paper/1807.08362/) (2018)
- [Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey](https://mlsanity.com/paper/2207.07068/) (2022)
- [Fairness risk measures](https://mlsanity.com/paper/1901.08665/) (2019)
- [An Empirical Study of Rich Subgroup Fairness for Machine Learning](https://mlsanity.com/paper/1808.08166/) (2018)
- [Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation](https://mlsanity.com/paper/2205.04610/) (2022)
- [R'enyi Fair Inference](https://mlsanity.com/paper/1906.12005/) (2019)
- [A Stochastic Optimization Framework for Fair Risk Minimization](https://mlsanity.com/paper/2102.12586/) (2021)
- [Faster Algorithms for Testing under Conditional Sampling](https://mlsanity.com/paper/1504.04103/) (2015)
- [A Model-Based Method for Minimizing CVaR and Beyond](https://mlsanity.com/paper/2305.17498/) (2023)
- [Fair Wrapping for Black-box Predictions](https://mlsanity.com/paper/2201.12947/) (2022)

## Built on by (most cited first, 3 of 3)

- [Multi-Group Proportional Representation in Retrieval](https://mlsanity.com/paper/2407.08571/) (2024)
- [When fairness metrics fail: A utility-based perspective on varepsilon-fairness](https://mlsanity.com/paper/2405.09360/) (2024)
- [FRAME: Separating sampling variation from performance disparities in medical image analysis](https://mlsanity.com/paper/2608.25981/) (2026)

## Abstract (arXiv)

Machine learning (ML) models used in prediction and classification tasks may display performance disparities across population groups determined by sensitive attributes (e.g., race, sex, age). We consider the problem of evaluating the performance of a fixed ML model across population groups defined by multiple sensitive attributes (e.g., race and sex and age). Here, the sample complexity for estimating the worst-case performance gap across groups (e.g., the largest difference in error rates) increases exponentially with the number of group-denoting sensitive attributes. To address this issue, we propose an approach to test for performance disparities based on Conditional Value-at-Risk (CVaR). By allowing a small probabilistic slack on the groups over which a model has approximately equal performance, we show that the sample complexity required for discovering performance violations is reduced exponentially to be at most upper bounded by the square root of the number of groups. As a byproduct of our analysis, when the groups are weighted by a specific prior distribution, we show that R'enyi entropy of order 2/3 of the prior distribution captures the sample complexity of the proposed CVaR test algorithm. Finally, we also show that there exists a non-i.i.d. data collection strategy that results in a sample complexity independent of the number of groups.