# ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learning

arXiv: https://arxiv.org/abs/2007.09339 · HTML: https://mlsanity.com/paper/2007.09339/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2020.

## Built on by (most cited first, 30 of 70)

- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Enhanced Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/2111.09679/) (2021)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks](https://mlsanity.com/paper/2203.03929/) (2022)
- [The Privacy Onion Effect: Memorization is Relative](https://mlsanity.com/paper/2206.10469/) (2022)
- [ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models](https://mlsanity.com/paper/2102.02551/) (2021)
- [Membership Inference of Diffusion Models](https://mlsanity.com/paper/2301.09956/) (2023)
- [Membership Inference on Word Embedding and Beyond](https://mlsanity.com/paper/2106.11384/) (2021)
- [Is Difficulty Calibration All We Need? Towards More Practical Membership Inference Attacks](https://mlsanity.com/paper/2409.00426/) (2024)
- [Membership Inference Attacks and Generalization: A Causal Perspective](https://mlsanity.com/paper/2209.08615/) (2022)
- [Adversarial Machine Unlearning](https://mlsanity.com/paper/2406.07687/) (2024)
- [De-identification is not enough: a comparison between de-identified and synthetic clinical notes](https://mlsanity.com/paper/2402.00179/) (2024)
- [Source Inference Attacks: Beyond Membership Inference Attacks in Federated Learning](https://mlsanity.com/paper/2310.00222/) (2023)
- [Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble](https://mlsanity.com/paper/2506.13972/) (2025)
- [SHAPr: An Efficient and Versatile Membership Privacy Risk Metric for Machine Learning](https://mlsanity.com/paper/2112.02230/) (2021)
- [Do Parameters Reveal More than Loss for Membership Inference?](https://mlsanity.com/paper/2406.11544/) (2024)
- [FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics](https://mlsanity.com/paper/2605.17373/) (2026)
- [Training Data Leakage Analysis in Language Models](https://mlsanity.com/paper/2101.05405/) (2021)
- [Improved Membership Inference Attacks Against Language Classification Models](https://mlsanity.com/paper/2310.07219/) (2023)
- [Free Record-Level Privacy Risk Evaluation Through Artifact-Based Methods](https://mlsanity.com/paper/2411.05743/) (2024)
- [Data Privacy and Trustworthy Machine Learning](https://mlsanity.com/paper/2209.06529/) (2022)
- [Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Development](https://mlsanity.com/paper/2307.11525/) (2023)
- [Cascading and Proxy Membership Inference Attacks](https://mlsanity.com/paper/2507.21412/) (2025)
- [Beyond Data Privacy: New Privacy Risks for Large Language Models](https://mlsanity.com/paper/2509.14278/) (2025)
- [FedLAP-DP: Federated Learning by Sharing Differentially Private Loss Approximations](https://mlsanity.com/paper/2302.01068/) (2023)
- [Little is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning](https://mlsanity.com/paper/2310.05696/) (2023)
- [Fundamental Limits of Membership Inference Attacks on Machine Learning Models](https://mlsanity.com/paper/2310.13786/) (2023)
- [Curvature Clues: Decoding Deep Learning Privacy with Input Loss Curvature](https://mlsanity.com/paper/2407.02747/) (2024)

## Abstract (arXiv)

When building machine learning models using sensitive data, organizations should ensure that the data processed in such systems is adequately protected. For projects involving machine learning on personal data, Article 35 of the GDPR mandates it to perform a Data Protection Impact Assessment (DPIA). In addition to the threats of illegitimate access to data through security breaches, machine learning models pose an additional privacy risk to the data by indirectly revealing about it through the model predictions and parameters. Guidances released by the Information Commissioner's Office (UK) and the National Institute of Standards and Technology (US) emphasize on the threat to data from models and recommend organizations to account for and estimate these risks to comply with data protection regulations. Hence, there is an immediate need for a tool that can quantify the privacy risk to data from models. In this paper, we focus on this indirect leakage about training data from machine learning models. We present ML Privacy Meter, a tool that can quantify the privacy risk to data from models through state of the art membership inference attack techniques. We discuss how this tool can help practitioners in compliance with data protection regulations, when deploying machine learning models.