# Calibrating Practical Privacy Risks for Differentially Private Machine Learning

arXiv: https://arxiv.org/abs/2410.22673 · HTML: https://mlsanity.com/paper/2410.22673/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 16 of 16)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/1911.07135/) (2019)
- [The Privacy Onion Effect: Memorization is Relative](https://mlsanity.com/paper/2206.10469/) (2022)
- [SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning](https://mlsanity.com/paper/2212.10986/) (2022)
- [Property Inference Attacks on Convolutional Neural Networks: Influence and Implications of Target Model's Complexity](https://mlsanity.com/paper/2104.13061/) (2021)
- ["Excavating AI" Re-excavated: Debunking a Fallacious Account of the JAFFE Dataset](https://mlsanity.com/paper/2107.13998/) (2021)
- [LDP-Feat: Image Features with Local Differential Privacy](https://mlsanity.com/paper/2308.11223/) (2023)
- [GAN-based Domain Inference Attack](https://mlsanity.com/paper/2212.11810/) (2022)
- [Correlated Differential Privacy: Feature Selection in Machine Learning](https://mlsanity.com/paper/2010.03094/) (2020)
- [The Need for Interpretable Features: Motivation and Taxonomy](https://mlsanity.com/paper/2202.11748/) (2022)

## Abstract (arXiv)

Differential privacy quantifies privacy through the privacy budget ε, yet its practical interpretation is complicated by variations across models and datasets. Recent research on differentially private machine learning and membership inference has highlighted that with the same theoretical ε setting, the likelihood-ratio-based membership inference (LiRA) attacking success rate (ASR) may vary according to specific datasets and models, which might be a better indicator for evaluating real-world privacy risks. Inspired by this practical privacy measure, we study the approaches that can lower the attacking success rate to allow for more flexible privacy budget settings in model training. We find that by selectively suppressing privacy-sensitive features, we can achieve lower ASR values without compromising application-specific data utility. We use the SHAP and LIME model explainer to evaluate feature sensitivities and develop feature-masking strategies. Our findings demonstrate that the LiRA ASRM on model M can properly indicate the inherent privacy risk of a dataset for modeling, and it's possible to modify datasets to enable the use of larger theoretical ε settings to achieve equivalent practical privacy protection. We have conducted extensive experiments to show the inherent link between ASR and the dataset's privacy risk. By carefully selecting features to mask, we can preserve more data utility with equivalent practical privacy protection and relaxed ε settings. The implementation details are shared online at the provided GitHub URL urlhttps://anonymous.4open.science/r/On-sensitive-features-and-empirical-epsilon-lower-bounds-BF67/.