# "You Can't Fix What You Can't Measure": Privately Measuring Demographic Performance Disparities in Federated Learning

arXiv: https://arxiv.org/abs/2206.12183 · HTML: https://mlsanity.com/paper/2206.12183/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 78 references that are in the map.
- First posted on arXiv 24 Jun 2022.

## Builds on (most cited first, 30 of 35)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response](https://mlsanity.com/paper/1407.6981/) (2014)
- [Fairness without Demographics through Adversarially Reweighted Learning](https://mlsanity.com/paper/2006.13114/) (2020)
- [Applied Federated Learning: Improving Google Keyboard Query Suggestions](https://mlsanity.com/paper/1812.02903/) (2018)
- [Amplification by Shuffling: From Local to Central Differential Privacy via Anonymity](https://mlsanity.com/paper/1811.12469/) (2018)
- [Reducing Gender Bias in Abusive Language Detection](https://mlsanity.com/paper/1808.07231/) (2018)
- [FairFed: Enabling Group Fairness in Federated Learning](https://mlsanity.com/paper/2110.00857/) (2021)
- [Finite Sample Differentially Private Confidence Intervals](https://mlsanity.com/paper/1711.03908/) (2017)
- [Differentially Private Fair Learning](https://mlsanity.com/paper/1812.02696/) (2018)
- [The Privacy Blanket of the Shuffle Model](https://mlsanity.com/paper/1903.02837/) (2019)
- [Covariance-Aware Private Mean Estimation Without Private Covariance Estimation](https://mlsanity.com/paper/2106.13329/) (2021)
- [Differential Privacy-enabled Federated Learning for Sensitive Health Data](https://mlsanity.com/paper/1910.02578/) (2019)
- [Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition](https://mlsanity.com/paper/2002.10361/) (2020)
- [Blind Justice: Fairness with Encrypted Sensitive Attributes](https://mlsanity.com/paper/1806.03281/) (2018)
- [Optimal Algorithms for Mean Estimation under Local Differential Privacy](https://mlsanity.com/paper/2205.02466/) (2022)
- [Evaluating Fairness of Machine Learning Models Under Uncertain and Incomplete Information](https://mlsanity.com/paper/2102.08410/) (2021)
- [Improving on-device speaker verification using federated learning with privacy](https://mlsanity.com/paper/2008.02651/) (2020)
- [Collecting and Analyzing Data from Smart Device Users with Local Differential Privacy](https://mlsanity.com/paper/1606.05053/) (2016)
- [Federated Learning for Ranking Browser History Suggestions](https://mlsanity.com/paper/1911.11807/) (2019)
- [Awareness in Practice: Tensions in Access to Sensitive Attribute Data for Antidiscrimination](https://mlsanity.com/paper/1912.06171/) (2019)
- [A Federated Learning Framework for Healthcare IoT devices](https://mlsanity.com/paper/2005.05083/) (2020)
- [Bias in Automated Speaker Recognition](https://mlsanity.com/paper/2201.09486/) (2022)
- [Adaptive Sampling Strategies to Construct Equitable Training Datasets](https://mlsanity.com/paper/2202.01327/) (2022)
- [Multiaccurate Proxies for Downstream Fairness](https://mlsanity.com/paper/2107.04423/) (2021)

## Built on by (most cited first, 3 of 3)

- [Can Querying for Bias Leak Protected Attributes? Achieving Privacy With Smooth Sensitivity](https://mlsanity.com/paper/2211.02139/) (2022)
- [SoK: What Makes Private Learning Unfair?](https://mlsanity.com/paper/2501.14414/) (2025)
- [Federated Fairness without Access to Sensitive Groups](https://mlsanity.com/paper/2402.14929/) (2024)

## Abstract (arXiv)

As in traditional machine learning models, models trained with federated learning may exhibit disparate performance across demographic groups. Model holders must identify these disparities to mitigate undue harm to the groups. However, measuring a model's performance in a group requires access to information about group membership which, for privacy reasons, often has limited availability. We propose novel locally differentially private mechanisms to measure differences in performance across groups while protecting the privacy of group membership. To analyze the effectiveness of the mechanisms, we bound their error in estimating a disparity when optimized for a given privacy budget. Our results show that the error rapidly decreases for realistic numbers of participating clients, demonstrating that, contrary to what prior work suggested, protecting privacy is not necessarily in conflict with identifying performance disparities of federated models.