# Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification

arXiv: https://arxiv.org/abs/2301.11562 · HTML: https://mlsanity.com/paper/2301.11562/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 27 Jan 2023.

## Builds on (most cited first, 10 of 10)

- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [On Fairness and Calibration](https://mlsanity.com/paper/1709.02012/) (2017)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [On Wasserstein Two Sample Testing and Related Families of Nonparametric Tests](https://mlsanity.com/paper/1509.02237/) (2015)
- [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019)
- [Why Is My Classifier Discriminatory?](https://mlsanity.com/paper/1805.12002/) (2018)
- [Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning](https://mlsanity.com/paper/2202.05338/) (2022)
- [Empirical observation of negligible fairness-accuracy trade-offs in machine learning for public policy](https://mlsanity.com/paper/2012.02972/) (2020)
- [Emergent Unfairness in Algorithmic Fairness-Accuracy Trade-Off Research](https://mlsanity.com/paper/2102.01203/) (2021)

## Built on by (most cited first, 30 of 32)

- [Amazing Things Come From Having Many Good Models](https://mlsanity.com/paper/2407.04846/) (2024)
- [Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning](https://mlsanity.com/paper/2501.14959/) (2025)
- [Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research](https://mlsanity.com/paper/2412.06966/) (2024)
- [One Model Many Scores: Using Multiverse Analysis to Prevent Fairness Hacking and Evaluate the Influence of Model Design Decisions](https://mlsanity.com/paper/2308.16681/) (2023)
- [Be Intentional About Fairness!: Fairness, Size, and Multiplicity in the Rashomon Set](https://mlsanity.com/paper/2501.15634/) (2025)
- [ARTICLE: Annotator Reliability Through In-Context Learning](https://mlsanity.com/paper/2409.12218/) (2024)
- [The Cost of Arbitrariness for Individuals: Examining the Legal and Technical Challenges of Model Multiplicity](https://mlsanity.com/paper/2407.13070/) (2024)
- [The Costs of Pretending That There Are Data-Generating Probability Distributions in the Social World](https://mlsanity.com/paper/2407.17395/) (2024)
- [Fairness and Sparsity within Rashomon sets: Enumeration-Free Exploration and Characterization](https://mlsanity.com/paper/2502.05286/) (2025)
- [The Role of Hyperparameters in Predictive Multiplicity](https://mlsanity.com/paper/2503.13506/) (2025)
- [FairlyUncertain: A Comprehensive Benchmark of Uncertainty in Algorithmic Fairness](https://mlsanity.com/paper/2410.02005/) (2024)
- [The Disparate Benefits of Deep Ensembles](https://mlsanity.com/paper/2410.13831/) (2024)
- [You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Models](https://mlsanity.com/paper/2410.20265/) (2024)
- [Underrepresentation, Label Bias, and Proxies: Towards Data Bias Profiles for the EU AI Act and Beyond](https://mlsanity.com/paper/2507.08866/) (2025)
- [Bias Begins with Data: The FairGround Corpus for Robust and Reproducible Research on Algorithmic Fairness](https://mlsanity.com/paper/2510.22363/) (2025)
- [Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity](https://mlsanity.com/paper/2602.00723/) (2026)
- [Using predictive multiplicity to measure individual performance within the AI Act](https://mlsanity.com/paper/2602.11944/) (2026)
- [Unified Approach for Weakly Supervised Multicalibration](https://mlsanity.com/paper/2605.09857/) (2026)
- [Predictive Multiplicity in Survival Models: A Method for Quantifying Model Uncertainty in Predictive Maintenance Applications](https://mlsanity.com/paper/2504.12156/) (2025)
- [Why are there many equally good models? An Anatomy of the Rashomon Effect](https://mlsanity.com/paper/2601.06730/) (2026)
- [Robust Ambiguity Detection (RAD) From Model- and Feature-Space Consistency](https://mlsanity.com/paper/2608.11541/) (2026)
- [Disparate Model Performance and Stability in Machine Learning Clinical Support for Diabetes and Heart Diseases](https://mlsanity.com/paper/2412.19495/) (2024)
- [Simple and Effective Specialized Representations for Fair Classifiers](https://mlsanity.com/paper/2505.11740/) (2025)
- [Unequal Uncertainty: Rethinking Algorithmic Interventions for Mitigating Discrimination from AI](https://mlsanity.com/paper/2508.07872/) (2025)
- [From Fragile to Certified: Wasserstein Audits of Group Fairness Under Distribution Shift](https://mlsanity.com/paper/2509.26241/) (2025)
- [Statistical Guarantees in the Search for Less Discriminatory Algorithms](https://mlsanity.com/paper/2512.23943/) (2025)
- [An empirical evaluation of the risks of AI model updates using clinical data: stability, arbitrariness, and fairness](https://mlsanity.com/paper/2604.23954/) (2026)
- [Trustworthy AI: Ensuring Reliability and Accountability from Models to Agents](https://mlsanity.com/paper/2605.08964/) (2026)
- [When Interpretability Is Unequally Distributed: Fairness in Hybrid Interpretable Models](https://mlsanity.com/paper/2605.28626/) (2026)
- [Model Multiplicity and Predictive Arbitrariness in Recidivism Risk Assessment](https://mlsanity.com/paper/2606.02198/) (2026)

## Abstract (arXiv)

Variance in predictions across different trained models is a significant, under-explored source of error in fair binary classification. In practice, the variance on some data examples is so large that decisions can be effectively arbitrary. To investigate this problem, we take an experimental approach and make four overarching contributions: We: 1) Define a metric called self-consistency, derived from variance, which we use as a proxy for measuring and reducing arbitrariness; 2) Develop an ensembling algorithm that abstains from classification when a prediction would be arbitrary; 3) Conduct the largest to-date empirical study of the role of variance (vis-a-vis self-consistency and arbitrariness) in fair binary classification; and, 4) Release a toolkit that makes the US Home Mortgage Disclosure Act (HMDA) datasets easily usable for future research. Altogether, our experiments reveal shocking insights about the reliability of conclusions on benchmark datasets. Most fair binary classification benchmarks are close-to-fair when taking into account the amount of arbitrariness present in predictions -- before we even try to apply any fairness interventions. This finding calls into question the practical utility of common algorithmic fairness methods, and in turn suggests that we should reconsider how we choose to measure fairness in binary classification.