# SynQP: A Framework and Metrics for Evaluating the Quality and Privacy Risk of Synthetic Data

arXiv: https://arxiv.org/abs/2601.12124 · HTML: https://mlsanity.com/paper/2601.12124/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 17 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI](https://mlsanity.com/paper/2204.01075/) (2022)
- [How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy](https://mlsanity.com/paper/2303.00654/) (2023)
- [Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations](https://mlsanity.com/paper/1910.08902/) (2019)
- [REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers](https://mlsanity.com/paper/2302.02041/) (2023)
- [Membership Inference Attacks against Synthetic Data through Overfitting Detection](https://mlsanity.com/paper/2302.12580/) (2023)
- [A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models](https://mlsanity.com/paper/2404.14445/) (2024)
- [A Consensus Privacy Metrics Framework for Synthetic Data](https://mlsanity.com/paper/2503.04980/) (2025)

## Abstract (arXiv)

The use of synthetic data in health applications raises privacy concerns, yet the lack of open frameworks for privacy evaluations has slowed its adoption. A major challenge is the absence of accessible benchmark datasets for evaluating privacy risks, due to difficulties in acquiring sensitive data. To address this, we introduce SynQP, an open framework for benchmarking privacy in synthetic data generation (SDG) using simulated sensitive data, ensuring that original data remains confidential. We also highlight the need for privacy metrics that fairly account for the probabilistic nature of machine learning models. As a demonstration, we use SynQP to benchmark CTGAN and propose a new identity disclosure risk metric that offers a more accurate estimation of privacy risks compared to existing approaches. Our work provides a critical tool for improving the transparency and reliability of privacy evaluations, enabling safer use of synthetic data in health-related applications. % In our quality evaluations, non-private models achieved near-perfect machine-learning efficacy (≥0.97). Our privacy assessments (Table II) reveal that DP consistently lowers both identity disclosure risk (SD-IDR) and membership-inference attack risk (SD-MIA), with all DP-augmented models staying below the 0.09 regulatory threshold. Code available at https://github.com/CAN-SYNH/SynQP