# Cherry on the Cake: Fairness is NOT an Optimization Problem

arXiv: https://arxiv.org/abs/2406.16606 · HTML: https://mlsanity.com/paper/2406.16606/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [On Fairness and Calibration](https://mlsanity.com/paper/1709.02012/) (2017)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [Fairness in Machine Learning: Lessons from Political Philosophy](https://mlsanity.com/paper/1712.03586/) (2017)
- [Classification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees](https://mlsanity.com/paper/1806.06055/) (2018)
- ['It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions](https://mlsanity.com/paper/1801.10408/) (2018)
- [Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AI](https://mlsanity.com/paper/2005.05906/) (2020)
- [Fairlearn: Assessing and Improving Fairness of AI Systems](https://mlsanity.com/paper/2303.16626/) (2023)
- [Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making](https://mlsanity.com/paper/1802.01029/) (2018)
- [Fairness without the sensitive attribute via Causal Variational Autoencoder](https://mlsanity.com/paper/2109.04999/) (2021)
- [Enforcing Group Fairness in Algorithmic Decision Making: Utility Maximization Under Sufficiency](https://mlsanity.com/paper/2206.02237/) (2022)
- [The Possibility of Fairness: Revisiting the Impossibility Theorem in Practice](https://mlsanity.com/paper/2302.06347/) (2023)
- [How to be fair? A study of label and selection bias](https://mlsanity.com/paper/2403.14282/) (2024)
- [A Discrete and Bounded Envy-Free Cake Cutting Protocol for Any Number of Agents](https://mlsanity.com/paper/1604.03655/) (2016)
- [Are There Exceptions to Goodhart's Law? On the Moral Justification of Fairness-Aware Machine Learning](https://mlsanity.com/paper/2202.08536/) (2022)

## Built on by (most cited first, 2 of 2)

- [Fairness vs Performance: Characterizing the Pareto Frontier of Algorithmic Decision Systems](https://mlsanity.com/paper/2605.10604/) (2026)
- [No evaluation without fair representation : Impact of label and selection bias on the evaluation, performance and mitigation of classification models](https://mlsanity.com/paper/2603.09662/) (2026)

## Abstract (arXiv)

In Fair AI literature, the practice of maliciously creating unfair models that nevertheless satisfy fairness constraints is known as "cherry-picking". A cherry-picking model is a model that makes mistakes on purpose, selecting bad individuals from a minority class instead of better candidates from the same minority. The model literally cherry-picks whom to select to superficially meet the fairness constraints while making minimal changes to the unfair model. This practice has been described as "blatantly unfair" and has a negative impact on already marginalized communities, undermining the intended purpose of fairness measures specifically designed to protect these communities. A common assumption is that cherry-picking arises solely from malicious intent and that models designed only to optimize fairness metrics would avoid this behavior. We show that this is not the case: models optimized to minimize fairness metrics while maximizing performance are often forced to cherry-pick to some degree. In other words, cherry-picking might be an inevitable outcome of the optimization process itself. To demonstrate this, we use tools from fair cake-cutting, a mathematical subfield that studies the problem of fairly dividing a resource, referred to as the "cake," among a number of participants. This concept is connected to supervised multi-label classification: any dataset can be thought of as a cake that needs to be distributed among different labels, and the model is the function that divides the cake. We adapt these classical results for machine learning and demonstrate how this connection can be prolifically used for fairness and classification in general.