# Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

arXiv: https://arxiv.org/abs/1709.01604 · HTML: https://mlsanity.com/paper/1709.01604/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2017.

## Built on by (most cited first, 13 of 13)

- [How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy](https://mlsanity.com/paper/2303.00654/) (2023)
- [Survey: Leakage and Privacy at Inference Time](https://mlsanity.com/paper/2107.01614/) (2021)
- [Differentially Private Decoding in Large Language Models](https://mlsanity.com/paper/2205.13621/) (2022)
- [Understanding Rare Spurious Correlations in Neural Networks](https://mlsanity.com/paper/2202.05189/) (2022)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [Privacy Assessment of Federated Learning using Private Personalized Layers](https://mlsanity.com/paper/2106.08060/) (2021)
- [Accurate, Explainable, and Private Models: Providing Recourse While Minimizing Training Data Leakage](https://mlsanity.com/paper/2308.04341/) (2023)
- [Anti-Neuron Watermarking: Protecting Personal Data Against Unauthorized Neural Networks](https://mlsanity.com/paper/2109.09023/) (2021)
- [Differentially Private Integrated Decision Gradients (IDG-DP) for Radar-based Human Activity Recognition](https://mlsanity.com/paper/2411.02099/) (2024)
- [Privacy Re-identification Attacks on Tabular GANs](https://mlsanity.com/paper/2404.00696/) (2024)
- [Machine Unlearning in Contrastive Learning](https://mlsanity.com/paper/2405.07317/) (2024)
- [PSY: Posterior Sampling Based Privacy Enhancer in Large Language Models](https://mlsanity.com/paper/2410.18824/) (2024)
- [An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information](https://mlsanity.com/paper/2610.09206/) (2026)

## Abstract (arXiv)

Machine learning algorithms, when applied to sensitive data, pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through the models' structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role. This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks and begins to shed light on what other factors may be in play. Finally, we explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks.