mlsanityData to 7 Oct 2026

Uncertainty-based Fairness Measures

arXiv 2312.11299 ↗First posted 18 Dec 2023cs.LG cs.CY stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

Unfair predictions of machine learning (ML) models impede their broad acceptance in real-world settings. Tackling this arduous challenge first necessitates defining what it means for an ML model to be fair. This has been addressed by the ML community with various measures of fairness that depend on the prediction outcomes of the ML models, either at the group level or the individual level. These fairness measures are limited in that they utilize point predictions, neglecting their variances, or uncertainties, making them susceptible to noise, missingness and shifts in data. In this paper, we first show that an ML model may appear to be fair with existing point-based fairness measures but biased against a demographic group in terms of prediction uncertainties. Then, we introduce new fairness measures based on different types of uncertainties, namely, aleatoric uncertainty and epistemic uncertainty. We demonstrate on many datasets that (i) our uncertainty-based measures are complementary to existing measures of fairness, and (ii) they provide more insights about the underlying issues leading to bias.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 43 of its 66 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

43 of 66 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  6. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  7. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  8. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  9. 2017Counterfactual Fairnesscited by 802
  10. 2014Certifying and removing disparate impactcited by 715
  11. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  12. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
31 more
  1. 2015Weight Uncertainty in Neural Networkscited by 554
  2. 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 443
  3. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  4. 2017Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairnesscited by 381
  5. 2016Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatmentcited by 368
  6. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  7. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  8. 2020Fairness in Machine Learningcited by 194
  9. 2020Training independent subnetworks for robust predictioncited by 177
  10. 2019Identifying and Correcting Label Bias in Machine Learningcited by 146
  11. 2022Bias Mitigation for Machine Learning Classifiers: A Comprehensive Surveycited by 125
  12. 2021A Clarification of the Nuances in the Fairness Metrics Landscapecited by 98
  13. 2020Investigating Bias and Fairness in Facial Expression Recognitioncited by 87
  14. 2020Fair Classification with Group-Dependent Label Noisecited by 67
  15. 2020Two Simple Ways to Learn Individual Fairness Metrics from Datacited by 54
  16. 2019A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inferencecited by 51
  17. 2021Understanding and Mitigating Annotation Bias in Facial Expression Recognitioncited by 50
  18. 2020Fairness Metrics: A Comparative Analysiscited by 45
  19. 2020Uncertainty Estimation Using a Single Deep Deterministic Neural Networkcited by 37
  20. 2022Fairness Transferability Subject to Bounded Distribution Shiftcited by 28
  21. 2021Responsible AI: Gender bias assessment in emotion recognitioncited by 22
  22. 2023Aleatoric and Epistemic Discrimination: Fundamental Limits of Fairness Interventionscited by 19
  23. 2020The Importance of Modeling Data Missingness in Algorithmic Fairness: A Causal Perspectivecited by 17
  24. 2023Fairness through Aleatoric Uncertaintycited by 16
  25. 2023Evaluating the Fairness of Deep Learning Uncertainty Estimates in Medical Image Analysiscited by 15
  26. 2020Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Boundscited by 13
  27. 2022Certifying Some Distributional Fairness with Subpopulation Decompositioncited by 11
  28. 2022Bias Reducing Multitask Learning on Mental Health Predictioncited by 9
  29. 2022Uncertainty-aware predictive modeling for fair data-driven decisionscited by 6
  30. 2023Fairness Uncertainty Quantification: How certain are you that the model is fair?cited by 4
  31. 2023Class Uncertainty: A Measure to Mitigate Class Imbalancecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 1.

15 papers.

Sort:

  1. 2025U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detectioncited by 12
  2. 2025Machine Learning Fairness for Depression Detection using EEG Datacited by 7
  3. 2025Gender Fairness of Machine Learning Algorithms for Pain Detectioncited by 3
  4. 2025Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolutioncited by 3
  5. 2025Fair Uncertainty Quantification for Depression Predictioncited by 2
  6. 2025Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMscited by 2
  7. 2026Fairness under uncertainty in sequential decisionscited by 2
  8. 2024FairlyUncertain: A Comprehensive Benchmark of Uncertainty in Algorithmic Fairnesscited by 1
  9. 2025Beyond Accuracy: What Matters in Designing Well-Behaved Image Classification Models?cited by 1
  10. 2025Achieving Distributive Justice in Federated Learning via Uncertainty Quantificationcited by 1
  11. 2025Size-adaptive Hypothesis Testing for Fairnesscited by 1
  12. 2025Automatic Depression Assessment using Machine Learning: A Comprehensive Surveycited by 1
3 more
  1. 2025FairSSL: Fair Multimodal Self-Supervised Learningcited by 1
  2. 2025FG 2025 TrustFAA: the First Workshop on Towards Trustworthy Facial Affect Analysis: Advancing Insights of Fairness, Explainability, and Safety (TrustFAA)cited by 0
  3. 2025Who Owns The Robot?: Four Ethical and Socio-technical Questions about Wellbeing Robots in the Real World through Community Engagementcited by 0