# Utility-Fairness Trade-Offs and How to Find Them

arXiv: https://arxiv.org/abs/2404.09454 · HTML: https://mlsanity.com/paper/2404.09454/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Learning Adversarially Fair and Transferable Representations](https://mlsanity.com/paper/1802.06309/) (2018)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Avoiding Discrimination through Causal Reasoning](https://mlsanity.com/paper/1706.02744/) (2017)
- [Controllable Invariance through Adversarial Feature Learning](https://mlsanity.com/paper/1705.11122/) (2017)
- [Mitigating Face Recognition Bias via Group Adaptive Classifier](https://mlsanity.com/paper/2006.07576/) (2020)
- [Mitigating Information Leakage in Image Representations: A Maximum Entropy Approach](https://mlsanity.com/paper/1904.05514/) (2019)
- [FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs](https://mlsanity.com/paper/2403.15593/) (2024)
- [Learning Unbiased Representations via R'enyi Minimization](https://mlsanity.com/paper/2009.03183/) (2020)
- [On the Global Optima of Kernelized Adversarial Representation Learning](https://mlsanity.com/paper/1910.07423/) (2019)
- [On Characterizing the Trade-off in Invariant Representation Learning](https://mlsanity.com/paper/2109.03386/) (2021)
- [Adversarial Representation Learning With Closed-Form Solvers](https://mlsanity.com/paper/2109.05535/) (2021)

## Built on by (most cited first, 15 of 15)

- [Joint Vision-Language Social Bias Removal for CLIP](https://mlsanity.com/paper/2411.12785/) (2024)
- [Fairness and Bias Mitigation in Computer Vision: A Survey](https://mlsanity.com/paper/2408.02464/) (2024)
- [OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes](https://mlsanity.com/paper/2501.00962/) (2025)
- [Efficient Fairness-Performance Pareto Front Computation](https://mlsanity.com/paper/2409.17643/) (2024)
- [Minority Collective Action for User-Side Fairness](https://mlsanity.com/paper/2508.15374/) (2025)
- [The Statistical Fairness-Accuracy Frontier](https://mlsanity.com/paper/2508.17622/) (2025)
- [From Global to Local: Social Bias Transfer in CLIP](https://mlsanity.com/paper/2508.17750/) (2025)
- [Obliviator Reveals the Cost of Nonlinear Guardedness in Concept Erasure](https://mlsanity.com/paper/2603.07529/) (2026)
- [A Multi-Objective Evaluation Framework for Analyzing Utility-Fairness Trade-Offs in Machine Learning Systems](https://mlsanity.com/paper/2503.11120/) (2025)
- [PolyJuice Makes It Real: Black-Box, Universal Red Teaming for Synthetic Image Detectors](https://mlsanity.com/paper/2509.15551/) (2025)
- [FairMT: Fairness for Heterogeneous Multi-Task Learning](https://mlsanity.com/paper/2512.00469/) (2025)
- [DoubleCCA: Improving Foundation Model Group Robustness with Random Sentence Embeddings](https://mlsanity.com/paper/2411.16236/) (2024)
- [Patient-Conditioned Adaptive Offsets for Reliable Diagnosis across Subgroups](https://mlsanity.com/paper/2601.13094/) (2026)
- [Federated Learning at the Forefront of Fairness: A Multifaceted Perspective](https://mlsanity.com/paper/2602.00718/) (2026)
- [FoeGlass: Simple In-Context Learning Is Enough for Red Teaming Audio Deepfake Detectors](https://mlsanity.com/paper/2606.05101/) (2026)

## Abstract (arXiv)

When building classification systems with demographic fairness considerations, there are two objectives to satisfy: 1) maximizing utility for the specific task and 2) ensuring fairness w.r.t. a known demographic attribute. These objectives often compete, so optimizing both can lead to a trade-off between utility and fairness. While existing works acknowledge the trade-offs and study their limits, two questions remain unanswered: 1) What are the optimal trade-offs between utility and fairness? and 2) How can we numerically quantify these trade-offs from data for a desired prediction task and demographic attribute of interest? This paper addresses these questions. We introduce two utility-fairness trade-offs: the Data-Space and Label-Space Trade-off. The trade-offs reveal three regions within the utility-fairness plane, delineating what is fully and partially possible and impossible. We propose U-FaTE, a method to numerically quantify the trade-offs for a given prediction task and group fairness definition from data samples. Based on the trade-offs, we introduce a new scheme for evaluating representations. An extensive evaluation of fair representation learning methods and representations from over 1000 pre-trained models revealed that most current approaches are far from the estimated and achievable fairness-utility trade-offs across multiple datasets and prediction tasks.