# Emergent Unfairness in Algorithmic Fairness-Accuracy Trade-Off Research

arXiv: https://arxiv.org/abs/2102.01203 · HTML: https://mlsanity.com/paper/2102.01203/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 76 references that are in the map.
- First posted on arXiv 1 Feb 2021.

## Builds on (most cited first, 14 of 14)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Fairness in Machine Learning: Lessons from Political Philosophy](https://mlsanity.com/paper/1712.03586/) (2017)
- [Inherent Tradeoffs in Learning Fair Representations](https://mlsanity.com/paper/1906.08386/) (2019)
- [Why Is My Classifier Discriminatory?](https://mlsanity.com/paper/1805.12002/) (2018)
- [Problem Formulation and Fairness](https://mlsanity.com/paper/1901.02547/) (2019)
- [Mathematical Notions vs. Human Perception of Fairness: A Descriptive Approach to Fairness for Machine Learning](https://mlsanity.com/paper/1902.04783/) (2019)
- [Roles for Computing in Social Change](https://mlsanity.com/paper/1912.04883/) (2019)
- [Fairness Behind a Veil of Ignorance: A Welfare Analysis for Automated Decision Making](https://mlsanity.com/paper/1806.04959/) (2018)
- [Active Fairness in Algorithmic Decision Making](https://mlsanity.com/paper/1810.00031/) (2018)
- [On the (im)possibility of fairness](https://mlsanity.com/paper/1609.07236/) (2016)

## Built on by (most cited first, 30 of 34)

- [Bias Testing and Mitigation in LLM-based Code Generation](https://mlsanity.com/paper/2309.14345/) (2023)
- [Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning](https://mlsanity.com/paper/2202.05338/) (2022)
- [Queer In AI: A Case Study in Community-Led Participatory AI](https://mlsanity.com/paper/2303.16972/) (2023)
- [Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification](https://mlsanity.com/paper/2301.11562/) (2023)
- [Factoring the Matrix of Domination: A Critical Review and Reimagination of Intersectionality in AI Fairness](https://mlsanity.com/paper/2303.17555/) (2023)
- [Hyperparameter Optimization Is Deceiving Us, and How to Stop It](https://mlsanity.com/paper/2102.03034/) (2021)
- [Ethical Considerations for Responsible Data Curation](https://mlsanity.com/paper/2302.03629/) (2023)
- [Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML](https://mlsanity.com/paper/2303.08485/) (2023)
- [Don't Throw it Away! The Utility of Unlabeled Data in Fair Decision Making](https://mlsanity.com/paper/2205.04790/) (2022)
- [U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detection](https://mlsanity.com/paper/2501.09687/) (2025)
- [How to be fair? A study of label and selection bias](https://mlsanity.com/paper/2403.14282/) (2024)
- [AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development](https://mlsanity.com/paper/2502.18682/) (2025)
- [Survey on Fairness Notions and Related Tensions](https://mlsanity.com/paper/2209.13012/) (2022)
- [Gradual (In)Compatibility of Fairness Criteria](https://mlsanity.com/paper/2109.04399/) (2021)
- [What Constitutes a Less Discriminatory Algorithm?](https://mlsanity.com/paper/2412.18138/) (2024)
- [Optimization's Neglected Normative Commitments](https://mlsanity.com/paper/2305.17465/) (2023)
- [A Catalog of Fairness-Aware Practices in Machine Learning Engineering](https://mlsanity.com/paper/2408.16683/) (2024)
- [A Justice-Based Framework for the Analysis of Algorithmic Fairness-Utility Trade-Offs](https://mlsanity.com/paper/2206.02891/) (2022)
- [Distributive Justice as the Foundational Premise of Fair ML: Unification, Extension, and Interpretation of Group Fairness Metrics](https://mlsanity.com/paper/2206.02897/) (2022)
- [A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapes](https://mlsanity.com/paper/2410.13286/) (2024)
- [The Statistical Fairness-Accuracy Frontier](https://mlsanity.com/paper/2508.17622/) (2025)
- [Fair Enough? A map of the current limitations of the requirements to have fair algorithms](https://mlsanity.com/paper/2311.12435/) (2023)
- [Causality Is Key to Understand and Balance Multiple Goals in Trustworthy ML and Foundation Models](https://mlsanity.com/paper/2502.21123/) (2025)
- [Aggregating Concepts of Fairness and Accuracy in Prediction Algorithms](https://mlsanity.com/paper/2505.08829/) (2025)
- [A multidomain relational framework to guide institutional AI research and adoption](https://mlsanity.com/paper/2303.10106/) (2023)
- [A Counterfactual Fair Model for Longitudinal Electronic Health Records via Deconfounder](https://mlsanity.com/paper/2308.11819/) (2023)
- [Biased Heritage: How Datasets Shape Models in Facial Expression Recognition](https://mlsanity.com/paper/2503.03446/) (2025)
- [From Utilitarian to Rawlsian Designs for Algorithmic Fairness](https://mlsanity.com/paper/2302.03567/) (2023)
- [The Flawed Foundations of Fair Machine Learning](https://mlsanity.com/paper/2306.01417/) (2023)
- [Towards Fairness in Provably Communication-Efficient Federated Recommender Systems](https://mlsanity.com/paper/2405.15788/) (2024)

## Abstract (arXiv)

Across machine learning (ML) sub-disciplines, researchers make explicit mathematical assumptions in order to facilitate proof-writing. We note that, specifically in the area of fairness-accuracy trade-off optimization scholarship, similar attention is not paid to the normative assumptions that ground this approach. Such assumptions presume that 1) accuracy and fairness are in inherent opposition to one another, 2) strict notions of mathematical equality can adequately model fairness, 3) it is possible to measure the accuracy and fairness of decisions independent from historical context, and 4) collecting more data on marginalized individuals is a reasonable solution to mitigate the effects of the trade-off. We argue that such assumptions, which are often left implicit and unexamined, lead to inconsistent conclusions: While the intended goal of this work may be to improve the fairness of machine learning models, these unexamined, implicit assumptions can in fact result in emergent unfairness. We conclude by suggesting a concrete path forward toward a potential resolution.