# A Consensus Privacy Metrics Framework for Synthetic Data

arXiv: https://arxiv.org/abs/2503.04980 · HTML: https://mlsanity.com/paper/2503.04980/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 138 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Builds on (most cited first, 29 of 29)

- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022)
- [Generating Multi-label Discrete Patient Records using Generative Adversarial Networks](https://mlsanity.com/paper/1703.06490/) (2017)
- [Data Synthesis based on Generative Adversarial Networks](https://mlsanity.com/paper/1806.03384/) (2018)
- [Synthetic Data -- Anonymisation Groundhog Day](https://mlsanity.com/paper/2011.07018/) (2020)
- [Membership Inference Attacks against Synthetic Data through Overfitting Detection](https://mlsanity.com/paper/2302.12580/) (2023)
- [TAPAS: a Toolbox for Adversarial Privacy Auditing of Synthetic Data](https://mlsanity.com/paper/2211.06550/) (2022)
- [Generating Synthetic Mixed-type Longitudinal Electronic Health Records for Artificial Intelligent Applications](https://mlsanity.com/paper/2112.12047/) (2021)
- [A Unified Framework for Quantifying Privacy Risk in Synthetic Data](https://mlsanity.com/paper/2211.10459/) (2022)
- [Geographic Spines in the 2020 Census Disclosure Avoidance System](https://mlsanity.com/paper/2203.16654/) (2022)
- [A Multifaceted Benchmarking of Synthetic Electronic Health Record Generation Models](https://mlsanity.com/paper/2208.01230/) (2022)
- [CorGAN: Correlation-Capturing Convolutional Generative Adversarial Networks for Generating Synthetic Healthcare Records](https://mlsanity.com/paper/2001.09346/) (2020)
- [Differentially Private Synthetic Data: Applied Evaluations and Enhancements](https://mlsanity.com/paper/2011.05537/) (2020)
- [Achilles' Heels: Vulnerable Record Identification in Synthetic Data Publishing](https://mlsanity.com/paper/2306.10308/) (2023)
- [Can I trust my fake data -- A comprehensive quality assessment framework for synthetic tabular data in healthcare](https://mlsanity.com/paper/2401.13716/) (2024)
- [Differentially Private Release of Israel's National Registry of Live Births](https://mlsanity.com/paper/2405.00267/) (2024)
- [Fidelity and Privacy of Synthetic Medical Data](https://mlsanity.com/paper/2101.08658/) (2021)
- [A review of Generative Adversarial Networks for Electronic Health Records: applications, evaluation measures and data sources](https://mlsanity.com/paper/2203.07018/) (2022)
- [Performance evaluation of predictive AI models to support medical decisions: Overview and guidance](https://mlsanity.com/paper/2412.10288/) (2024)
- [On Utility and Privacy in Synthetic Genomic Data](https://mlsanity.com/paper/2102.03314/) (2021)
- [30 Years of Synthetic Data](https://mlsanity.com/paper/2304.02107/) (2023)
- [An Empirical Study on the Membership Inference Attack against Tabular Data Synthesis Models](https://mlsanity.com/paper/2208.08114/) (2022)
- [A Members First Approach to Enabling LinkedIn's Labor Market Insights at Scale](https://mlsanity.com/paper/2010.13981/) (2020)
- [Database Reconstruction Is Not So Easy and Is Different from Reidentification](https://mlsanity.com/paper/2301.10213/) (2023)
- [Privacy Measurement in Tabular Synthetic Data: State of the Art and Future Research Directions](https://mlsanity.com/paper/2311.17453/) (2023)
- [A Note on the Misinterpretation of the US Census Re-identification Attack](https://mlsanity.com/paper/2202.04872/) (2022)
- [Selecting a classification performance measure: matching the measure to the problem](https://mlsanity.com/paper/2409.12391/) (2024)
- [Towards more accurate and useful data anonymity vulnerability measures](https://mlsanity.com/paper/2403.06595/) (2024)
- [Synthetic Data and Health Privacy](https://mlsanity.com/paper/2501.09031/) (2025)

## Built on by (most cited first, 8 of 8)

- [Privacy-Preserving Tabular Synthetic Data Generation Using TabularARGN](https://mlsanity.com/paper/2508.06647/) (2025)
- [Synthetic data for ratemaking: imputation-based methods vs adversarial networks and autoencoders](https://mlsanity.com/paper/2509.02171/) (2025)
- [SynQP: A Framework and Metrics for Evaluating the Quality and Privacy Risk of Synthetic Data](https://mlsanity.com/paper/2601.12124/) (2026)
- [Evaluation metrics for temporal preservation in synthetic longitudinal patient data](https://mlsanity.com/paper/2602.10643/) (2026)
- [A Critical Review on the Effectiveness and Privacy Threats of Membership Inference Attacks](https://mlsanity.com/paper/2603.22987/) (2026)
- [Not all Jensen-Shannon Divergence Estimators are Equal](https://mlsanity.com/paper/2606.16411/) (2026)
- [Generating Benchmark Health Data Using a Tabular Diffusion Transformer](https://mlsanity.com/paper/2608.14496/) (2026)
- [Position: Privacy Is a Claim, Not a Property of Synthetic Data](https://mlsanity.com/paper/2609.01273/) (2026)

## Abstract (arXiv)

Synthetic data generation is one approach for sharing individual-level data. However, to meet legislative requirements, it is necessary to demonstrate that the individuals' privacy is adequately protected. There is no consolidated standard for measuring privacy in synthetic data. Through an expert panel and consensus process, we developed a framework for evaluating privacy in synthetic data. Our findings indicate that current similarity metrics fail to measure identity disclosure, and their use is discouraged. For differentially private synthetic data, a privacy budget other than close to zero was not considered interpretable. There was consensus on the importance of membership and attribute disclosure, both of which involve inferring personal information about an individual without necessarily revealing their identity. The resultant framework provides precise recommendations for metrics that address these types of disclosures effectively. Our findings further present specific opportunities for future research that can help with widespread adoption of synthetic data.