# Formalizing and Estimating Distribution Inference Risks

arXiv: https://arxiv.org/abs/2109.06024 · HTML: https://mlsanity.com/paper/2109.06024/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Detecting AI Trojans Using Meta Neural Analysis](https://mlsanity.com/paper/1910.03137/) (2019)
- [Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers](https://mlsanity.com/paper/1306.4447/) (2013)
- [Stealing Hyperparameters in Machine Learning](https://mlsanity.com/paper/1802.05351/) (2018)
- [Dataset Inference: Ownership Resolution in Machine Learning](https://mlsanity.com/paper/2104.10706/) (2021)
- [Unleashing the Tiger: Inference Attacks on Split Learning](https://mlsanity.com/paper/2012.02670/) (2020)
- [Property Inference From Poisoning](https://mlsanity.com/paper/2101.11073/) (2021)
- [Leakage of Dataset Properties in Multi-Party Machine Learning](https://mlsanity.com/paper/2006.07267/) (2020)
- [Survey: Leakage and Privacy at Inference Time](https://mlsanity.com/paper/2107.01614/) (2021)
- [Disparate Vulnerability to Membership Inference Attacks](https://mlsanity.com/paper/1906.00389/) (2019)
- [SoK: Differential Privacies](https://mlsanity.com/paper/1906.01337/) (2019)
- [Automating Botnet Detection with Graph Neural Networks](https://mlsanity.com/paper/2003.06344/) (2020)
- [Attribute Privacy: Framework and Mechanisms](https://mlsanity.com/paper/2009.04013/) (2020)
- [A Novel Attribute Reconstruction Attack in Federated Learning](https://mlsanity.com/paper/2108.06910/) (2021)

## Built on by (most cited first, 30 of 43)

- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Reconstructing Training Data with Informed Adversaries](https://mlsanity.com/paper/2201.04845/) (2022)
- [SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning](https://mlsanity.com/paper/2212.10986/) (2022)
- [SoK: Memorization in General-Purpose Large Language Models](https://mlsanity.com/paper/2310.18362/) (2023)
- [User Inference Attacks on Large Language Models](https://mlsanity.com/paper/2310.09266/) (2023)
- [SNAP: Efficient Extraction of Private Properties with Poisoning](https://mlsanity.com/paper/2208.12348/) (2022)
- [On Linear Representations and Pretraining Data Frequency in Language Models](https://mlsanity.com/paper/2504.12459/) (2025)
- [Group Property Inference Attacks Against Graph Neural Networks](https://mlsanity.com/paper/2209.01100/) (2022)
- [Subject Membership Inference Attacks in Federated Learning](https://mlsanity.com/paper/2206.03317/) (2022)
- [Distribution inference risks: Identifying and mitigating sources of leakage](https://mlsanity.com/paper/2209.08541/) (2022)
- [GrOVe: Ownership Verification of Graph Neural Networks using Embeddings](https://mlsanity.com/paper/2304.08566/) (2023)
- [Dissecting Distribution Inference](https://mlsanity.com/paper/2212.07591/) (2022)
- [Inf2Guard: An Information-Theoretic Framework for Learning Privacy-Preserving Representations against Inference Attacks](https://mlsanity.com/paper/2403.02116/) (2024)
- [Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?](https://mlsanity.com/paper/2407.16607/) (2024)
- [Range Membership Inference Attacks](https://mlsanity.com/paper/2408.05131/) (2024)
- [Protecting Global Properties of Datasets with Distribution Privacy Mechanisms](https://mlsanity.com/paper/2207.08367/) (2022)
- [Manipulating Transfer Learning for Property Inference](https://mlsanity.com/paper/2303.11643/) (2023)
- [Correlation inference attacks against machine learning models](https://mlsanity.com/paper/2112.08806/) (2021)
- [Struct-Bench: A Benchmark for Differentially Private Structured Text Generation](https://mlsanity.com/paper/2509.10696/) (2025)
- [Summary Statistic Privacy in Data Sharing](https://mlsanity.com/paper/2303.02014/) (2023)
- [SoK: Unintended Interactions among Machine Learning Defenses and Risks](https://mlsanity.com/paper/2312.04542/) (2023)
- [SafeNet: The Unreasonable Effectiveness of Ensembles in Private Collaborative Learning](https://mlsanity.com/paper/2205.09986/) (2022)
- [Black-Box Audits for Group Distribution Shifts](https://mlsanity.com/paper/2209.03620/) (2022)
- [Attesting Distributional Properties of Training Data for Machine Learning](https://mlsanity.com/paper/2308.09552/) (2023)
- [Chameleon: Increasing Label-Only Membership Leakage with Adaptive Poisoning](https://mlsanity.com/paper/2310.03838/) (2023)
- [SoK: Data Reconstruction Attacks Against Machine Learning Models: Definition, Metrics, and Benchmark](https://mlsanity.com/paper/2506.07888/) (2025)
- [PriSampler: Mitigating Property Inference of Diffusion Models](https://mlsanity.com/paper/2306.05208/) (2023)
- [Transpose Attack: Stealing Datasets with Bidirectional Training](https://mlsanity.com/paper/2311.07389/) (2023)
- [Analyzing Inference Privacy Risks Through Gradients in Machine Learning](https://mlsanity.com/paper/2408.16913/) (2024)
- [Toward Efficient Inference Attacks: Shadow Model Sharing via Mixture-of-Experts](https://mlsanity.com/paper/2510.13451/) (2025)

## Abstract (arXiv)

Distribution inference, sometimes called property inference, infers statistical properties about a training set from access to a model trained on that data. Distribution inference attacks can pose serious risks when models are trained on private data, but are difficult to distinguish from the intrinsic purpose of statistical machine learning -- namely, to produce models that capture statistical properties about a distribution. Motivated by Yeom et al.'s membership inference framework, we propose a formal definition of distribution inference attacks that is general enough to describe a broad class of attacks distinguishing between possible training distributions. We show how our definition captures previous ratio-based property inference attacks as well as new kinds of attack including revealing the average node degree or clustering coefficient of a training graph. To understand distribution inference risks, we introduce a metric that quantifies observed leakage by relating it to the leakage that would occur if samples from the training distribution were provided directly to the adversary. We report on a series of experiments across a range of different distributions using both novel black-box attacks and improved versions of the state-of-the-art white-box attacks. Our results show that inexpensive attacks are often as effective as expensive meta-classifier attacks, and that there are surprising asymmetries in the effectiveness of attacks. Code is available at https://github.com/iamgroot42/FormEstDistRisks