# Synthetic Data Privacy Metrics

arXiv: https://arxiv.org/abs/2501.03941 · HTML: https://mlsanity.com/paper/2501.03941/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 95 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Builds on (most cited first, 30 of 50)

- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Auditing Differentially Private Machine Learning: How Private is Private SGD?](https://mlsanity.com/paper/2006.07709/) (2020)
- [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](https://mlsanity.com/paper/2101.04535/) (2021)
- [Generating Multi-label Discrete Patient Records using Generative Adversarial Networks](https://mlsanity.com/paper/1703.06490/) (2017)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space](https://mlsanity.com/paper/2310.09656/) (2023)
- [Data Synthesis based on Generative Adversarial Networks](https://mlsanity.com/paper/1806.03384/) (2018)
- [Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks](https://mlsanity.com/paper/2203.03929/) (2022)
- [Membership Leakage in Label-Only Exposures](https://mlsanity.com/paper/2007.15528/) (2020)
- [CTAB-GAN: Effective Table Data Synthesizing](https://mlsanity.com/paper/2102.08369/) (2021)
- [On the Importance of Difficulty Calibration in Membership Inference Attacks](https://mlsanity.com/paper/2111.08440/) (2021)
- [The Privacy Onion Effect: Memorization is Relative](https://mlsanity.com/paper/2206.10469/) (2022)
- [Are Diffusion Models Vulnerable to Membership Inference Attacks?](https://mlsanity.com/paper/2302.01316/) (2023)
- [Privacy Auditing with One (1) Training Run](https://mlsanity.com/paper/2305.08846/) (2023)
- [Reconstruction and Membership Inference Attacks against Generative Models](https://mlsanity.com/paper/1906.03006/) (2019)
- [Revisiting Membership Inference Under Realistic Assumptions](https://mlsanity.com/paper/2005.10881/) (2020)
- [Synthetic Data -- Anonymisation Groundhog Day](https://mlsanity.com/paper/2011.07018/) (2020)

## Built on by (most cited first, 8 of 8)

- [Graph-Conditional Flow Matching for Relational Data Generation](https://mlsanity.com/paper/2505.15668/) (2025)
- [Membership Inference Attack with Partial Features](https://mlsanity.com/paper/2508.06244/) (2025)
- [ICAFS: Inter-Client-Aware Feature Selection for Vertical Federated Learning](https://mlsanity.com/paper/2504.10851/) (2025)
- [Limited Reference, Reliable Generation: A Two-Component Framework for Tabular Data Generation in Low-Data Regimes](https://mlsanity.com/paper/2509.09960/) (2025)
- [Toward Valid Generative Clinical Trial Data with Survival Endpoints](https://mlsanity.com/paper/2511.16551/) (2025)
- [Controllable Generative Sandbox for Causal Inference](https://mlsanity.com/paper/2603.03587/) (2026)
- [Evaluating LLM Simulators as Differentially Private Data Generators](https://mlsanity.com/paper/2604.15461/) (2026)
- [FaceKit: a Toolkit for Interpretable Facial Phenotyping, Synthetic Image Generation and Privacy Analysis in Rare Diseases](https://mlsanity.com/paper/2610.09288/) (2026)

## Abstract (arXiv)

Recent advancements in generative AI have made it possible to create synthetic datasets that can be as accurate as real-world data for training AI models, powering statistical insights, and fostering collaboration with sensitive datasets while offering strong privacy guarantees. Effectively measuring the empirical privacy of synthetic data is an important step in the process. However, while there is a multitude of new privacy metrics being published every day, there currently is no standardization. In this paper, we review the pros and cons of popular metrics that include simulations of adversarial attacks. We also review current best practices for amending generative models to enhance the privacy of the data they create (e.g. differential privacy).