# Can Querying for Bias Leak Protected Attributes? Achieving Privacy With Smooth Sensitivity arXiv: https://arxiv.org/abs/2211.02139 ยท HTML: https://mlsanity.com/paper/2211.02139/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 79 references that are in the map. - First posted on arXiv 3 Nov 2022. ## Builds on (most cited first, 30 of 30) - [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016) - [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019) - [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014) - [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017) - [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016) - [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021) - [Differential Privacy Has Disparate Impact on Model Accuracy](https://mlsanity.com/paper/1905.12101/) (2019) - [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017) - [No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems](https://mlsanity.com/paper/2011.12945/) (2020) - [Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices](https://mlsanity.com/paper/1906.09208/) (2019) - [Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved](https://mlsanity.com/paper/1811.11154/) (2018) - [Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination](https://mlsanity.com/paper/1906.00285/) (2019) - [Differentially Private Fair Learning](https://mlsanity.com/paper/1812.02696/) (2018) - [When Machine Learning Meets Privacy: A Survey and Outlook](https://mlsanity.com/paper/2011.11819/) (2020) - [iFair: Learning Individually Fair Data Representations for Algorithmic Decision Making](https://mlsanity.com/paper/1806.01059/) (2018) - [Operationalizing Individual Fairness with Pairwise Fair Representations](https://mlsanity.com/paper/1907.01439/) (2019) - [Sparse Modeling for Image and Vision Processing](https://mlsanity.com/paper/1411.3230/) (2014) - [Blind Justice: Fairness with Encrypted Sensitive Attributes](https://mlsanity.com/paper/1806.03281/) (2018) - [An Overview of Privacy in Machine Learning](https://mlsanity.com/paper/2005.08679/) (2020) - [Proxy Fairness](https://mlsanity.com/paper/1806.11212/) (2018) - [Active Fairness Auditing](https://mlsanity.com/paper/2206.08450/) (2022) - [Fairness Under Feature Exemptions: Counterfactual and Observational Measures](https://mlsanity.com/paper/2006.07986/) (2020) - [Fair lending needs explainable models for responsible recommendation](https://mlsanity.com/paper/1809.04684/) (2018) - [A Distributed Fair Machine Learning Framework with Private Demographic Data Protection](https://mlsanity.com/paper/1909.08081/) (2019) - [Use Privacy in Data-Driven Systems: Theory and Experiments with Machine Learnt Programs](https://mlsanity.com/paper/1705.07807/) (2017) - [Statistical Properties of Sanitized Results from Differentially Private Laplace Mechanism with Univariate Bounding Constraints](https://mlsanity.com/paper/1607.08554/) (2016) - ["You Can't Fix What You Can't Measure": Privately Measuring Demographic Performance Disparities in Federated Learning](https://mlsanity.com/paper/2206.12183/) (2022) - [Compressed Sensing Using Binary Matrices of Nearly Optimal Dimensions](https://mlsanity.com/paper/1808.03001/) (2018) - [CryptoCredit: Securely Training Fair Models](https://mlsanity.com/paper/2010.04840/) (2020) - [The Cost of a Reductions Approach to Private Fair Optimization](https://mlsanity.com/paper/1906.09613/) (2019) ## Built on by (most cited first, 8 of 8) - [FairProof : Confidential and Certifiable Fairness for Neural Networks](https://mlsanity.com/paper/2402.12572/) (2024) - [Trained Random Forests Completely Reveal your Dataset](https://mlsanity.com/paper/2402.19232/) (2024) - [Demystifying Local and Global Fairness Trade-offs in Federated Learning Using Partial Information Decomposition](https://mlsanity.com/paper/2307.11333/) (2023) - [Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning](https://mlsanity.com/paper/2410.12085/) (2024) - [Probabilistic Dataset Reconstruction from Interpretable Models](https://mlsanity.com/paper/2308.15099/) (2023) - [A Unified View of Group Fairness Tradeoffs Using Partial Information Decomposition](https://mlsanity.com/paper/2406.04562/) (2024) - [Privacy Constrained Fairness Estimation for Decision Trees](https://mlsanity.com/paper/2312.08413/) (2023) - [Training Set Reconstruction from Differentially Private Forests: How Effective is DP?](https://mlsanity.com/paper/2502.05307/) (2025) ## Abstract (arXiv) Existing regulations prohibit model developers from accessing protected attributes (gender, race, etc.), often resulting in fairness assessments on populations without knowing their protected groups. In such scenarios, institutions often adopt a separation between the model developers (who train models with no access to the protected attributes) and a compliance team (who may have access to the entire dataset for auditing purposes). However, the model developers might be allowed to test their models for bias by querying the compliance team for group fairness metrics. In this paper, we first demonstrate that simply querying for fairness metrics, such as statistical parity and equalized odds can leak the protected attributes of individuals to the model developers. We demonstrate that there always exist strategies by which the model developers can identify the protected attribute of a targeted individual in the test dataset from just a single query. In particular, we show that one can reconstruct the protected attributes of all the individuals from O(Nk log( n /Nk)) queries when Nk<