# Distribution inference risks: Identifying and mitigating sources of leakage

arXiv: https://arxiv.org/abs/2209.08541 · HTML: https://mlsanity.com/paper/2209.08541/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 18 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Causal inference using invariant prediction: identification and confidence intervals](https://mlsanity.com/paper/1501.01332/) (2015)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Causality for Machine Learning](https://mlsanity.com/paper/1911.10500/) (2019)
- [R'enyi Differential Privacy of the Sampled Gaussian Mechanism](https://mlsanity.com/paper/1908.10530/) (2019)
- [Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers](https://mlsanity.com/paper/1306.4447/) (2013)
- [Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution](https://mlsanity.com/paper/1801.04016/) (2018)
- [Property Inference From Poisoning](https://mlsanity.com/paper/2101.11073/) (2021)
- [Leakage of Dataset Properties in Multi-Party Machine Learning](https://mlsanity.com/paper/2006.07267/) (2020)
- [Formalizing and Estimating Distribution Inference Risks](https://mlsanity.com/paper/2109.06024/) (2021)
- [SoK: Differential Privacies](https://mlsanity.com/paper/1906.01337/) (2019)
- [Property Inference Attacks Against GANs](https://mlsanity.com/paper/2111.07608/) (2021)
- [Towards Measuring Membership Privacy](https://mlsanity.com/paper/1712.09136/) (2017)
- [Subject Membership Inference Attacks in Federated Learning](https://mlsanity.com/paper/2206.03317/) (2022)
- [Alleviating Privacy Attacks via Causal Learning](https://mlsanity.com/paper/1909.12732/) (2019)
- [Towards Causal Federated Learning For Enhanced Robustness and Privacy](https://mlsanity.com/paper/2104.06557/) (2021)

## Built on by (most cited first, 17 of 17)

- [SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning](https://mlsanity.com/paper/2212.10986/) (2022)
- [SoK: Memorization in General-Purpose Large Language Models](https://mlsanity.com/paper/2310.18362/) (2023)
- [User Inference Attacks on Large Language Models](https://mlsanity.com/paper/2310.09266/) (2023)
- [The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration](https://mlsanity.com/paper/2509.14284/) (2025)
- [Dissecting Distribution Inference](https://mlsanity.com/paper/2212.07591/) (2022)
- [Inf2Guard: An Information-Theoretic Framework for Learning Privacy-Preserving Representations against Inference Attacks](https://mlsanity.com/paper/2403.02116/) (2024)
- [Manipulating Transfer Learning for Property Inference](https://mlsanity.com/paper/2303.11643/) (2023)
- [SoK: Unintended Interactions among Machine Learning Defenses and Risks](https://mlsanity.com/paper/2312.04542/) (2023)
- [Attesting Distributional Properties of Training Data for Machine Learning](https://mlsanity.com/paper/2308.09552/) (2023)
- [Can We Infer Confidential Properties of Training Data from LLMs?](https://mlsanity.com/paper/2506.10364/) (2025)
- [Subject Data Auditing via Source Inference Attack in Cross-Silo Federated Learning](https://mlsanity.com/paper/2409.19417/) (2024)
- [Meta-Statistical Learning: Supervised Learning of Statistical Estimators](https://mlsanity.com/paper/2502.12088/) (2025)
- [FLMarket: Enabling Privacy-preserved Pre-training Data Pricing for Federated Learning](https://mlsanity.com/paper/2411.11713/) (2024)
- [Unraveling Attacks in Machine Learning-based IoT Ecosystems: A Survey and the Open Libraries Behind Them](https://mlsanity.com/paper/2401.11723/) (2024)
- [Black-Box Privacy Attacks on Shared Representations in Multitask Learning](https://mlsanity.com/paper/2506.16460/) (2025)
- [Fair Finetuning Mitigates Distribution Inference Attacks](https://mlsanity.com/paper/2606.01719/) (2026)
- [QuanText: Protecting Dataset-Level Secrets in Textual Data Sharing](https://mlsanity.com/paper/2609.17995/) (2026)

## Abstract (arXiv)

A large body of work shows that machine learning (ML) models can leak sensitive or confidential information about their training data. Recently, leakage due to distribution inference (or property inference) attacks is gaining attention. In this attack, the goal of an adversary is to infer distributional information about the training data. So far, research on distribution inference has focused on demonstrating successful attacks, with little attention given to identifying the potential causes of the leakage and to proposing mitigations. To bridge this gap, as our main contribution, we theoretically and empirically analyze the sources of information leakage that allows an adversary to perpetrate distribution inference attacks. We identify three sources of leakage: (1) memorizing specific information about the mathbbE[Y|X] (expected label given the feature values) of interest to the adversary, (2) wrong inductive bias of the model, and (3) finiteness of the training data. Next, based on our analysis, we propose principled mitigation techniques against distribution inference attacks. Specifically, we demonstrate that causal learning techniques are more resilient to a particular type of distribution inference risk termed distributional membership inference than associative learning methods. And lastly, we present a formalization of distribution inference that allows for reasoning about more general adversaries than was previously possible.