# Dataset Representativeness and Downstream Task Fairness

arXiv: https://arxiv.org/abs/2407.00170 · HTML: https://mlsanity.com/paper/2407.00170/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 52 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [Simple data balancing achieves competitive worst-group-accuracy](https://mlsanity.com/paper/2110.14503/) (2021)
- [Why Is My Classifier Discriminatory?](https://mlsanity.com/paper/1805.12002/) (2018)
- [Bandits with concave rewards and convex knapsacks](https://mlsanity.com/paper/1402.5758/) (2014)
- [A Comparative Analysis of XGBoost](https://mlsanity.com/paper/1911.01914/) (2019)
- [Improving the Fairness of Chest X-ray Classifiers](https://mlsanity.com/paper/2203.12609/) (2022)
- [Representation Bias in Data: A Survey on Identification and Resolution Techniques](https://mlsanity.com/paper/2203.11852/) (2022)
- [Active Sampling for Min-Max Fairness](https://mlsanity.com/paper/2006.06879/) (2020)
- [Overwriting Pretrained Bias with Finetuning Data](https://mlsanity.com/paper/2303.06167/) (2023)
- [Adaptive Sampling for Minimax Fair Classification](https://mlsanity.com/paper/2103.00755/) (2021)
- [Assessing and Remedying Coverage for a Given Dataset](https://mlsanity.com/paper/1810.06742/) (2018)
- [Online Learning with Vector Costs and Bandits with Knapsacks](https://mlsanity.com/paper/2010.07346/) (2020)
- [Decision trees as partitioning machines to characterize their generalization properties](https://mlsanity.com/paper/2010.07374/) (2020)
- [Achieving Representative Data via Convex Hull Feasibility Sampling Algorithms](https://mlsanity.com/paper/2204.06664/) (2022)

## Abstract (arXiv)

Our society collects data on people for a wide range of applications, from building a census for policy evaluation to running meaningful clinical trials. To collect data, we typically sample individuals with the goal of accurately representing a population of interest. However, current sampling processes often collect data opportunistically from data sources, which can lead to datasets that are biased and not representative, i.e., the collected dataset does not accurately reflect the distribution of demographics of the true population. This is a concern because subgroups within the population can be under- or over-represented in a dataset, which may harm generalizability and lead to an unequal distribution of benefits and harms from downstream tasks that use such datasets (e.g., algorithmic bias in medical decision-making algorithms). In this paper, we assess the relationship between dataset representativeness and group-fairness of classifiers trained on that dataset. We demonstrate that there is a natural tension between dataset representativeness and classifier fairness; empirically we observe that training datasets with better representativeness can frequently result in classifiers with higher rates of unfairness. We provide some intuition as to why this occurs via a set of theoretical results in the case of univariate classifiers. We also find that over-sampling underrepresented groups can result in classifiers which exhibit greater bias to those groups. Lastly, we observe that fairness-aware sampling strategies (i.e., those which are specifically designed to select data with high downstream fairness) will often over-sample members of majority groups. These results demonstrate that the relationship between dataset representativeness and downstream classifier fairness is complex; balancing these two quantities requires special care from both model- and dataset-designers.