# The Measure and Mismeasure of Fairness

arXiv: https://arxiv.org/abs/1808.00023 · HTML: https://mlsanity.com/paper/1808.00023/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2018.

## Built on by (most cited first, 30 of 95)

- [What's in a Name? Auditing Large Language Models for Race and Gender Bias](https://mlsanity.com/paper/2402.14875/) (2024)
- [Causal Fairness Analysis](https://mlsanity.com/paper/2207.11385/) (2022)
- [Evaluating language models as risk scores](https://mlsanity.com/paper/2407.14614/) (2024)
- [Auditing the Use of Language Models to Guide Hiring Decisions](https://mlsanity.com/paper/2404.03086/) (2024)
- [Zero-shot causal learning](https://mlsanity.com/paper/2301.12292/) (2023)
- [fairadapt: Causal Reasoning for Fair Data Pre-processing](https://mlsanity.com/paper/2110.10200/) (2021)
- [DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems](https://mlsanity.com/paper/2211.05764/) (2022)
- [The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models](https://mlsanity.com/paper/2510.23191/) (2025)
- [Learning to be Fair: A Consequentialist Approach to Equitable Decision-Making](https://mlsanity.com/paper/2109.08792/) (2021)
- [No Fair Lunch: A Causal Perspective on Dataset Bias in Machine Learning for Medical Imaging](https://mlsanity.com/paper/2307.16526/) (2023)
- [Equality of opportunity in travel behavior prediction with deep neural networks and discrete choice models](https://mlsanity.com/paper/2109.12422/) (2021)
- [Fairness-enhancing deep learning for ride-hailing demand prediction](https://mlsanity.com/paper/2303.05698/) (2023)
- [Counterfactual Fairness by Combining Factual and Counterfactual Predictions](https://mlsanity.com/paper/2409.01977/) (2024)
- [Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments](https://mlsanity.com/paper/2410.21131/) (2024)
- [On Prediction-Modelers and Decision-Makers: Why Fairness Requires More Than a Fair Prediction Model](https://mlsanity.com/paper/2310.05598/) (2023)
- [Explaining CLIP's performance disparities on data from blind/low vision users](https://mlsanity.com/paper/2311.17315/) (2023)
- [Local Causal Discovery for Structural Evidence of Direct Discrimination](https://mlsanity.com/paper/2405.14848/) (2024)
- [Why Representation Engineering Works: A Theoretical and Empirical Study in Vision-Language Models](https://mlsanity.com/paper/2503.22720/) (2025)
- [Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness](https://mlsanity.com/paper/2506.04193/) (2025)
- [The Intersectionality Problem for Algorithmic Fairness](https://mlsanity.com/paper/2411.02569/) (2024)
- [Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessing](https://mlsanity.com/paper/2501.06366/) (2025)
- [Overcoming Fairness Trade-offs via Pre-processing: A Causal Perspective](https://mlsanity.com/paper/2501.14710/) (2025)
- [AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data](https://mlsanity.com/paper/2503.05665/) (2025)
- [On the Current and Emerging Challenges of Developing Fair and Ethical AI Solutions in Financial Services](https://mlsanity.com/paper/2111.01306/) (2021)
- [Survey on AI Ethics: A Socio-technical Perspective](https://mlsanity.com/paper/2311.17228/) (2023)
- [AI-Driven Healthcare: A Review on Ensuring Fairness and Mitigating Bias](https://mlsanity.com/paper/2407.19655/) (2024)
- [TechOps: Technical Documentation Templates for the AI Act](https://mlsanity.com/paper/2508.08804/) (2025)
- [Designing Equitable Algorithms](https://mlsanity.com/paper/2302.09157/) (2023)
- [FairRR: Pre-Processing for Group Fairness through Randomized Response](https://mlsanity.com/paper/2403.07780/) (2024)
- [Debiasing surgeon: fantastic weights and how to find them](https://mlsanity.com/paper/2403.14200/) (2024)

## Abstract (arXiv)

The field of fair machine learning aims to ensure that decisions guided by algorithms are equitable. Over the last decade, several formal, mathematical definitions of fairness have gained prominence. Here we first assemble and categorize these definitions into two broad families: (1) those that constrain the effects of decisions on disparities; and (2) those that constrain the effects of legally protected characteristics, like race and gender, on decisions. We then show, analytically and empirically, that both families of definitions typically result in strongly Pareto dominated decision policies. For example, in the case of college admissions, adhering to popular formal conceptions of fairness would simultaneously result in lower student-body diversity and a less academically prepared class, relative to what one could achieve by explicitly tailoring admissions policies to achieve desired outcomes. In this sense, requiring that these fairness definitions hold can, perversely, harm the very groups they were designed to protect. In contrast to axiomatic notions of fairness, we argue that the equitable design of algorithms requires grappling with their context-specific consequences, akin to the equitable design of policy. We conclude by listing several open challenges in fair machine learning and offering strategies to ensure algorithms are better aligned with policy goals.