# Leave-one-out Unfairness

arXiv: https://arxiv.org/abs/2107.10171 · HTML: https://mlsanity.com/paper/2107.10171/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 67 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Provable defenses against adversarial examples via the convex outer adversarial polytope](https://mlsanity.com/paper/1711.00851/) (2017)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference](https://mlsanity.com/paper/1906.11798/) (2019)
- [Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices](https://mlsanity.com/paper/1906.09208/) (2019)
- [Machine Learning Models that Remember Too Much](https://mlsanity.com/paper/1709.07886/) (2017)
- [Fairness in Learning: Classic and Contextual Bandits](https://mlsanity.com/paper/1605.07139/) (2016)
- [Preserving Statistical Validity in Adaptive Data Analysis](https://mlsanity.com/paper/1411.2664/) (2014)
- [Data-Dependent Stability of Stochastic Gradient Descent](https://mlsanity.com/paper/1703.01678/) (2017)
- [On the Connection Between Adversarial Robustness and Saliency Map Interpretability](https://mlsanity.com/paper/1905.04172/) (2019)
- [Fairness Under Composition](https://mlsanity.com/paper/1806.06122/) (2018)
- [E.T.-RNN: Applying Deep Learning to Credit Loan Applications](https://mlsanity.com/paper/1911.02496/) (2019)

## Built on by (most cited first, 30 of 36)

- [Training Data Influence Analysis and Estimation: A Survey](https://mlsanity.com/paper/2212.04612/) (2022)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Predictive Multiplicity in Probabilistic Classification](https://mlsanity.com/paper/2206.01131/) (2022)
- [Consistent Counterfactuals for Deep Models](https://mlsanity.com/paper/2110.03109/) (2021)
- [Understanding Instance-Level Impact of Fairness Constraints](https://mlsanity.com/paper/2206.15437/) (2022)
- [Selective Ensembles for Consistent Predictions](https://mlsanity.com/paper/2111.08230/) (2021)
- [Certifying Robustness to Programmable Data Bias in Decision Trees](https://mlsanity.com/paper/2110.04363/) (2021)
- [Rank List Sensitivity of Recommender Systems to Interaction Perturbations](https://mlsanity.com/paper/2201.12686/) (2022)
- [The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions](https://mlsanity.com/paper/2304.10655/) (2023)
- [Adapting and Evaluating Influence-Estimation Methods for Gradient-Boosted Decision Trees](https://mlsanity.com/paper/2205.00359/) (2022)
- [Predictive Churn with the Set of Good Models](https://mlsanity.com/paper/2402.07745/) (2024)
- [Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints](https://mlsanity.com/paper/2306.13738/) (2023)
- [Be Intentional About Fairness!: Fairness, Size, and Multiplicity in the Rashomon Set](https://mlsanity.com/paper/2501.15634/) (2025)
- [Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective](https://mlsanity.com/paper/2401.04374/) (2024)
- [Towards Algorithmic Fairness by means of Instance-level Data Re-weighting based on Shapley Values](https://mlsanity.com/paper/2303.01928/) (2023)
- [Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools](https://mlsanity.com/paper/2309.17337/) (2023)
- [Fairness and Sequential Decision Making: Limits, Lessons, and Opportunities](https://mlsanity.com/paper/2301.05753/) (2023)
- [A Simple and Efficient Baseline for Data Attribution on Images](https://mlsanity.com/paper/2311.03386/) (2023)
- [Towards Effective Discrimination Testing for Generative AI](https://mlsanity.com/paper/2412.21052/) (2024)
- [Certifying Data-Bias Robustness in Linear Regression](https://mlsanity.com/paper/2206.03575/) (2022)
- [On Minimizing the Impact of Dataset Shifts on Actionable Explanations](https://mlsanity.com/paper/2306.06716/) (2023)
- [Different Horses for Different Courses: Comparing Bias Mitigation Algorithms in ML](https://mlsanity.com/paper/2411.11101/) (2024)
- [FINEST: Stabilizing Recommendations by Rank-Preserving Fine-Tuning](https://mlsanity.com/paper/2402.03481/) (2024)
- [Fast-DataShapley: Neural Modeling for Training Data Valuation](https://mlsanity.com/paper/2506.05281/) (2025)
- [Testing Most Influential Sets](https://mlsanity.com/paper/2510.20372/) (2025)
- [An Epistemic and Aleatoric Decomposition of Arbitrariness to Constrain the Set of Good Models](https://mlsanity.com/paper/2302.04525/) (2023)
- [Enhancing Interpretability in Generative AI Through Search-Based Data Influence Analysis](https://mlsanity.com/paper/2504.01771/) (2025)
- [Detect & Score: Privacy-Preserving Misbehaviour Detection and Contribution Evaluation in Federated Learning](https://mlsanity.com/paper/2506.23583/) (2025)
- [On Arbitrary Predictions from Equally Valid Models](https://mlsanity.com/paper/2507.19408/) (2025)
- [Training Data Attribution for Image Generation using Ontology-Aligned Knowledge Graphs](https://mlsanity.com/paper/2512.02713/) (2025)

## Abstract (arXiv)

We introduce leave-one-out unfairness, which characterizes how likely a model's prediction for an individual will change due to the inclusion or removal of a single other person in the model's training data. Leave-one-out unfairness appeals to the idea that fair decisions are not arbitrary: they should not be based on the chance event of any one person's inclusion in the training data. Leave-one-out unfairness is closely related to algorithmic stability, but it focuses on the consistency of an individual point's prediction outcome over unit changes to the training data, rather than the error of the model in aggregate. Beyond formalizing leave-one-out unfairness, we characterize the extent to which deep models behave leave-one-out unfairly on real data, including in cases where the generalization error is small. Further, we demonstrate that adversarial training and randomized smoothing techniques have opposite effects on leave-one-out fairness, which sheds light on the relationships between robustness, memorization, individual fairness, and leave-one-out fairness in deep models. Finally, we discuss salient practical applications that may be negatively affected by leave-one-out unfairness.