# SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning

arXiv: https://arxiv.org/abs/2212.10986 · HTML: https://mlsanity.com/paper/2212.10986/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 89 references that are in the map.
- First posted on arXiv 21 Dec 2022.

## Builds on (most cited first, 30 of 53)

- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Knockoff Nets: Stealing Functionality of Black-Box Models](https://mlsanity.com/paper/1812.02766/) (2018)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)
- [Auditing Differentially Private Machine Learning: How Private is Private SGD?](https://mlsanity.com/paper/2006.07709/) (2020)
- [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](https://mlsanity.com/paper/2101.04535/) (2021)
- [Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference](https://mlsanity.com/paper/1906.11798/) (2019)
- [Membership Leakage in Label-Only Exposures](https://mlsanity.com/paper/2007.15528/) (2020)
- [Reconstructing Training Data with Informed Adversaries](https://mlsanity.com/paper/2201.04845/) (2022)
- [A Survey of Privacy Attacks in Machine Learning](https://mlsanity.com/paper/2007.07646/) (2020)
- [Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers](https://mlsanity.com/paper/1306.4447/) (2013)
- [Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning](https://mlsanity.com/paper/1904.01067/) (2019)
- [Reconstruction and Membership Inference Attacks against Generative Models](https://mlsanity.com/paper/1906.03006/) (2019)
- [Stealing Hyperparameters in Machine Learning](https://mlsanity.com/paper/1802.05351/) (2018)
- [Revisiting Membership Inference Under Realistic Assumptions](https://mlsanity.com/paper/2005.10881/) (2020)
- [Synthetic Data -- Anonymisation Groundhog Day](https://mlsanity.com/paper/2011.07018/) (2020)
- [Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets](https://mlsanity.com/paper/2204.00032/) (2022)
- [Analyzing Information Leakage of Updates to Natural Language Models](https://mlsanity.com/paper/1912.07942/) (2019)
- [On the Privacy Risks of Algorithmic Fairness](https://mlsanity.com/paper/2011.03731/) (2020)
- [Practical Blind Membership Inference Attack via Differential Comparisons](https://mlsanity.com/paper/2101.01341/) (2021)
- [EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learning](https://mlsanity.com/paper/2108.11023/) (2021)
- [When Machine Learning Meets Privacy: A Survey and Outlook](https://mlsanity.com/paper/2011.11819/) (2020)
- [Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture](https://mlsanity.com/paper/2110.08324/) (2021)
- [Property Inference From Poisoning](https://mlsanity.com/paper/2101.11073/) (2021)
- [Membership Inference Attacks Against Recommender Systems](https://mlsanity.com/paper/2109.08045/) (2021)

## Built on by (most cited first, 30 of 51)

- [Membership Inference Attacks Against In-Context Learning](https://mlsanity.com/paper/2409.01380/) (2024)
- [SoK: Memorization in General-Purpose Large Language Models](https://mlsanity.com/paper/2310.18362/) (2023)
- [User Inference Attacks on Large Language Models](https://mlsanity.com/paper/2310.09266/) (2023)
- [Identifying and Mitigating Privacy Risks Stemming from Language Models: A Survey](https://mlsanity.com/paper/2310.01424/) (2023)
- [PreCurious: How Innocent Pre-Trained Language Models Turn into Privacy Traps](https://mlsanity.com/paper/2403.09562/) (2024)
- [The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text](https://mlsanity.com/paper/2502.14921/) (2025)
- [Reinforcement Unlearning](https://mlsanity.com/paper/2312.15910/) (2023)
- [Unifying Re-Identification, Attribute Inference, and Data Reconstruction Risks in Differential Privacy](https://mlsanity.com/paper/2507.06969/) (2025)
- [Inf2Guard: An Information-Theoretic Framework for Learning Privacy-Preserving Representations against Inference Attacks](https://mlsanity.com/paper/2403.02116/) (2024)
- [The DCR Delusion: Measuring the Privacy Risk of Synthetic Data](https://mlsanity.com/paper/2505.01524/) (2025)
- [The Mosaic Memory of Large Language Models](https://mlsanity.com/paper/2405.15523/) (2024)
- [Do Parameters Reveal More than Loss for Membership Inference?](https://mlsanity.com/paper/2406.11544/) (2024)
- [Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective](https://mlsanity.com/paper/2506.13009/) (2025)
- [SoK: Unintended Interactions among Machine Learning Defenses and Risks](https://mlsanity.com/paper/2312.04542/) (2023)
- [Gaussian DP for Reporting Differential Privacy Guarantees in Machine Learning](https://mlsanity.com/paper/2503.10945/) (2025)
- [VISTA: Vision-Language Inference for Training-Free Stock Time-Series Analysis](https://mlsanity.com/paper/2505.18570/) (2025)
- [State-of-the-Art Approaches to Enhancing Privacy Preservation of Machine Learning Datasets: A Survey](https://mlsanity.com/paper/2404.16847/) (2024)
- [SoK: Data Reconstruction Attacks Against Machine Learning Models: Definition, Metrics, and Benchmark](https://mlsanity.com/paper/2506.07888/) (2025)
- [The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditing](https://mlsanity.com/paper/2506.16666/) (2025)
- [How do we measure privacy in text? A survey of text anonymization metrics](https://mlsanity.com/paper/2512.01109/) (2025)
- [Analyzing Inference Privacy Risks Through Gradients in Machine Learning](https://mlsanity.com/paper/2408.16913/) (2024)
- [From Principle to Practice: Vertical Data Minimization for Machine Learning](https://mlsanity.com/paper/2311.10500/) (2023)
- [Publishing Neural Networks in Drug Discovery Might Compromise Training Data Privacy](https://mlsanity.com/paper/2410.16975/) (2024)
- [Learning Robust and Privacy-Preserving Representations via Information Theory](https://mlsanity.com/paper/2412.11066/) (2024)
- [ExpShield: Safeguarding Web Text from Unauthorized Crawling and LLM Exploitation](https://mlsanity.com/paper/2412.21123/) (2024)
- [Why Train More? Effective and Efficient Membership Inference via Memorization](https://mlsanity.com/paper/2310.08015/) (2023)
- [SoK: Security and Privacy Risks of Healthcare AI](https://mlsanity.com/paper/2409.07415/) (2024)
- [A Unified Framework for Adversary-Aware Differential Privacy Bounds](https://mlsanity.com/paper/2507.08158/) (2025)
- [Causal Evaluation of Membership Inference Attacks](https://mlsanity.com/paper/2602.02819/) (2026)
- [Ensembler: Protect Collaborative Inference Privacy from Model Inversion Attack via Selective Ensemble](https://mlsanity.com/paper/2401.10859/) (2024)

## Abstract (arXiv)

Deploying machine learning models in production may allow adversaries to infer sensitive information about training data. There is a vast literature analyzing different types of inference risks, ranging from membership inference to reconstruction attacks. Inspired by the success of games (i.e., probabilistic experiments) to study security properties in cryptography, some authors describe privacy inference risks in machine learning using a similar game-based style. However, adversary capabilities and goals are often stated in subtly different ways from one presentation to the other, which makes it hard to relate and compose results. In this paper, we present a game-based framework to systematize the body of knowledge on privacy inference risks in machine learning. We use this framework to (1) provide a unifying structure for definitions of inference risks, (2) formally establish known relations among definitions, and (3) to uncover hitherto unknown relations that would have been difficult to spot otherwise.