# Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference

arXiv: https://arxiv.org/abs/2010.09851 · HTML: https://mlsanity.com/paper/2010.09851/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2020.

## Built on by (most cited first, 30 of 31)

- [Counterfactual Explanations Can Be Manipulated](https://mlsanity.com/paper/2106.02666/) (2021)
- [Evaluation Gaps in Machine Learning Practice](https://mlsanity.com/paper/2205.05256/) (2022)
- [Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities](https://mlsanity.com/paper/2102.04257/) (2021)
- [A Sociotechnical View of Algorithmic Fairness](https://mlsanity.com/paper/2110.09253/) (2021)
- [E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing](https://mlsanity.com/paper/2512.03109/) (2025)
- [Bounding and Approximating Intersectional Fairness through Marginal Fairness](https://mlsanity.com/paper/2206.05828/) (2022)
- [A structured regression approach for evaluating model performance across intersectional subgroups](https://mlsanity.com/paper/2401.14893/) (2024)
- [Semi-Supervised Learning of Classifiers from a Statistical Perspective: A Brief Review](https://mlsanity.com/paper/2104.04046/) (2021)
- [Model-based metrics: Sample-efficient estimates of predictive model subpopulation performance](https://mlsanity.com/paper/2104.12231/) (2021)
- [Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools](https://mlsanity.com/paper/2309.17337/) (2023)
- [Shortcut Learning in Binary Classifier Black Boxes: Applications to Voice Anti-Spoofing and Biometrics](https://mlsanity.com/paper/2601.17782/) (2026)
- [Active Assessment of Prediction Services as Accuracy Surface Over Attribute Combinations](https://mlsanity.com/paper/2108.06514/) (2021)
- [Adapting Fairness Interventions to Missing Values](https://mlsanity.com/paper/2305.19429/) (2023)
- [Fairness Hacking: The Malicious Practice of Shrouding Unfairness in Algorithms](https://mlsanity.com/paper/2311.06826/) (2023)
- [Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspective](https://mlsanity.com/paper/2302.08287/) (2023)
- [SUDO: a framework for evaluating clinical artificial intelligence systems without ground-truth annotations](https://mlsanity.com/paper/2403.17011/) (2024)
- [Evaluating multiple models using labeled and unlabeled data](https://mlsanity.com/paper/2501.11866/) (2025)
- [Are fairness metric scores enough to assess discrimination biases in machine learning?](https://mlsanity.com/paper/2306.05307/) (2023)
- [Don't Lie to Me: Avoiding Malicious Explanations with STEALTH](https://mlsanity.com/paper/2301.10407/) (2023)
- [FairlyUncertain: A Comprehensive Benchmark of Uncertainty in Algorithmic Fairness](https://mlsanity.com/paper/2410.02005/) (2024)
- [Defuse: Harnessing Unrestricted Adversarial Examples for Debugging Models Beyond Test Accuracy](https://mlsanity.com/paper/2102.06162/) (2021)
- [Uncertainty in Fairness Assessment: Maintaining Stable Conclusions Despite Fluctuations](https://mlsanity.com/paper/2302.01079/) (2023)
- [A Note on Bias to Complete](https://mlsanity.com/paper/2402.11710/) (2024)
- [A Rapid Test for Accuracy and Bias of Face Recognition Technology](https://mlsanity.com/paper/2502.14996/) (2025)
- [CEREAL: Few-Sample Clustering Evaluation](https://mlsanity.com/paper/2210.00064/) (2022)
- [(Un)certainty of (Un)fairness: Preference-Based Selection of Certainly Fair Decision-Makers](https://mlsanity.com/paper/2409.12677/) (2024)
- [A Critical Review of Predominant Bias in Neural Networks](https://mlsanity.com/paper/2502.11031/) (2025)
- [Beyond Internal Data: Constructing Complete Datasets for Fairness Testing](https://mlsanity.com/paper/2507.18561/) (2025)
- [Beyond Internal Data: Bounding and Estimating Fairness from Incomplete Data](https://mlsanity.com/paper/2508.13040/) (2025)
- [From Fragile to Certified: Wasserstein Audits of Group Fairness Under Distribution Shift](https://mlsanity.com/paper/2509.26241/) (2025)

## Abstract (arXiv)

We investigate the problem of reliably assessing group fairness when labeled examples are few but unlabeled examples are plentiful. We propose a general Bayesian framework that can augment labeled data with unlabeled data to produce more accurate and lower-variance estimates compared to methods based on labeled data alone. Our approach estimates calibrated scores for unlabeled examples in each group using a hierarchical latent variable model conditioned on labeled examples. This in turn allows for inference of posterior distributions with associated notions of uncertainty for a variety of group fairness metrics. We demonstrate that our approach leads to significant and consistent reductions in estimation error across multiple well-known fairness datasets, sensitive attributes, and predictive models. The results show the benefits of using both unlabeled data and Bayesian inference in terms of assessing whether a prediction model is fair or not.