# Lazy Data Practices Harm Fairness Research

arXiv: https://arxiv.org/abs/2404.17293 · HTML: https://mlsanity.com/paper/2404.17293/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 107 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 30 of 32)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [A comparative study of fairness-enhancing interventions in machine learning](https://mlsanity.com/paper/1802.04422/) (2018)
- [Algorithmic Fairness](https://mlsanity.com/paper/2001.09784/) (2020)
- [Re-imagining Algorithmic Fairness in India and Beyond](https://mlsanity.com/paper/2101.09995/) (2021)
- [Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved](https://mlsanity.com/paper/1811.11154/) (2018)
- [An Empirical Study of Rich Subgroup Fairness for Machine Learning](https://mlsanity.com/paper/1808.08166/) (2018)
- [It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks](https://mlsanity.com/paper/2106.05498/) (2021)
- [Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation](https://mlsanity.com/paper/2205.04610/) (2022)
- ["What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairness](https://mlsanity.com/paper/2011.02282/) (2020)
- [The Forgotten Margins of AI Ethics](https://mlsanity.com/paper/2205.04221/) (2022)
- [Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey](https://mlsanity.com/paper/2309.13933/) (2023)
- [WEIRD FAccTs: How Western, Educated, Industrialized, Rich, and Democratic is FAccT?](https://mlsanity.com/paper/2305.06415/) (2023)
- [Toward Fairness in AI for People with Disabilities: A Research Roadmap](https://mlsanity.com/paper/1907.02227/) (2019)
- [AI Fairness for People with Disabilities: Point of View](https://mlsanity.com/paper/1811.10670/) (2018)
- [Augmented Datasheets for Speech Datasets and Ethical Decision-Making](https://mlsanity.com/paper/2305.04672/) (2023)
- [Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders](https://mlsanity.com/paper/2205.08928/) (2022)
- [Four Years of FAccT: A Reflexive, Mixed-Methods Analysis of Research Contributions, Shortcomings, and Future Prospects](https://mlsanity.com/paper/2206.06738/) (2022)
- [Demographic-Reliant Algorithmic Fairness: Characterizing the Risks of Demographic Data Collection in the Pursuit of Fairness](https://mlsanity.com/paper/2205.01038/) (2022)
- [Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis](https://mlsanity.com/paper/2206.06149/) (2022)
- [Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approach](https://mlsanity.com/paper/2109.08549/) (2021)
- [An Empirical Analysis of Racial Categories in the Algorithmic Fairness Literature](https://mlsanity.com/paper/2309.06607/) (2023)
- [What is the Point of Fairness? Disability, AI and The Complexity of Justice](https://mlsanity.com/paper/1908.01024/) (2019)
- [Fairness Implications of Encoding Protected Categorical Attributes](https://mlsanity.com/paper/2201.11358/) (2022)

## Built on by (most cited first, 10 of 10)

- [Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation](https://mlsanity.com/paper/2502.06559/) (2025)
- [FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation](https://mlsanity.com/paper/2410.17358/) (2024)
- [Bias Begins with Data: The FairGround Corpus for Robust and Reproducible Research on Algorithmic Fairness](https://mlsanity.com/paper/2510.22363/) (2025)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Fairness in Federated Learning: Fairness for Whom?](https://mlsanity.com/paper/2505.21584/) (2025)
- [Dual-Metric Evaluation of Social Bias in Large Language Models: Evidence from an Underrepresented Nepali Cultural Context](https://mlsanity.com/paper/2603.07792/) (2026)
- [Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making](https://mlsanity.com/paper/2602.11924/) (2026)
- [Bureaucratic Silences: What the Canadian AI Register Reveals, Omits, and Obscures](https://mlsanity.com/paper/2604.15514/) (2026)
- [Equity Bias: An Ethical Framework for AI Design](https://mlsanity.com/paper/2604.21907/) (2026)
- [Why AI Harms Can't Be Fixed One Identity at a Time: What 5300 Incident Reports Reveal About Intersectionality](https://mlsanity.com/paper/2604.24519/) (2026)

## Abstract (arXiv)

Data practices shape research and practice on fairness in machine learning (fair ML). Critical data studies offer important reflections and critiques for the responsible advancement of the field by highlighting shortcomings and proposing recommendations for improvement. In this work, we present a comprehensive analysis of fair ML datasets, demonstrating how unreflective yet common practices hinder the reach and reliability of algorithmic fairness findings. We systematically study protected information encoded in tabular datasets and their usage in 280 experiments across 142 publications. Our analyses identify three main areas of concern: (1) a lack of representation for certain protected attributes in both data and evaluations; (2) the widespread exclusion of minorities during data preprocessing; and (3) opaque data processing threatening the generalization of fairness research. By conducting exemplary analyses on the utilization of prominent datasets, we demonstrate how unreflective data decisions disproportionately affect minority groups, fairness metrics, and resultant model comparisons. Additionally, we identify supplementary factors such as limitations in publicly available data, privacy considerations, and a general lack of awareness, which exacerbate these challenges. To address these issues, we propose a set of recommendations for data usage in fairness research centered on transparency and responsible inclusion. This study underscores the need for a critical reevaluation of data practices in fair ML and offers directions to improve both the sourcing and usage of datasets.