# SynthEval: A Framework for Detailed Utility and Privacy Evaluation of Tabular Synthetic Data

arXiv: https://arxiv.org/abs/2404.15821 · HTML: https://mlsanity.com/paper/2404.15821/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 49 references that are in the map.
- First posted on arXiv 24 Apr 2024.

## Builds on (most cited first, 10 of 10)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [CTAB-GAN: Effective Table Data Synthesizing](https://mlsanity.com/paper/2102.08369/) (2021)
- [DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks](https://mlsanity.com/paper/2110.12884/) (2021)
- [Synthcity: facilitating innovative use cases of synthetic data in different data modalities](https://mlsanity.com/paper/2301.07573/) (2023)
- [A Multifaceted Benchmarking of Synthetic Electronic Health Record Generation Models](https://mlsanity.com/paper/2208.01230/) (2022)
- [Generating Electronic Health Records with Multiple Data Types and Constraints](https://mlsanity.com/paper/2003.07904/) (2020)
- [Relational Data Synthesis using Generative Adversarial Networks: A Design Space Exploration](https://mlsanity.com/paper/2008.12763/) (2020)
- [Permutation-Invariant Tabular Data Synthesis](https://mlsanity.com/paper/2211.09286/) (2022)
- [Deep generative models in DataSHIELD](https://mlsanity.com/paper/2003.07775/) (2020)

## Built on by (most cited first, 27 of 27)

- [Synthetic Tabular Data: Methods, Attacks and Defenses](https://mlsanity.com/paper/2506.06108/) (2025)
- [Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation -- Extended Version](https://mlsanity.com/paper/2501.09591/) (2025)
- [TabRep: Training Tabular Diffusion Models with a Simple and Effective Continuous Representation](https://mlsanity.com/paper/2504.04798/) (2025)
- [Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis](https://mlsanity.com/paper/2503.02152/) (2025)
- [Cascaded Flow Matching for Heterogeneous Tabular Data with Mixed-Type Features](https://mlsanity.com/paper/2601.22816/) (2026)
- [Improving TabPFN's Synthetic Data Generation by Integrating Causal Structure](https://mlsanity.com/paper/2603.10254/) (2026)
- [TABFAIRGDT: A Fast Fair Tabular Data Generator using Autoregressive Decision Trees](https://mlsanity.com/paper/2509.19927/) (2025)
- [TabQueryBench: A Query-Centric Benchmark for Synthetic Tabular Data](https://mlsanity.com/paper/2607.03926/) (2026)
- [Privacy-Preserving Fair Synthetic Tabular Data](https://mlsanity.com/paper/2503.02968/) (2025)
- [SoK: Can Synthetic Images Replace Real Data? A Survey of Utility and Privacy of Synthetic Image Generation](https://mlsanity.com/paper/2506.19360/) (2025)
- [Privacy-Preserving Synthetic Review Generation with Diverse Writing Styles Using LLMs](https://mlsanity.com/paper/2507.18055/) (2025)
- [Disjoint Generation of Synthetic Data](https://mlsanity.com/paper/2507.19700/) (2025)
- [Synthetic Homes: A Multimodal Generative AI Pipeline for Residential Building Data Generation under Data Scarcity](https://mlsanity.com/paper/2509.09794/) (2025)
- [Measuring LLM Sensitivity in Transformer-based Tabular Data Synthesis](https://mlsanity.com/paper/2509.20768/) (2025)
- [Generating High-quality Privacy-preserving Synthetic Data](https://mlsanity.com/paper/2602.06390/) (2026)
- [Quantifying Membership Disclosure Risk for Tabular Synthetic Data Using Kernel Density Estimators](https://mlsanity.com/paper/2603.10937/) (2026)
- [Synthetic Tabular Generators Fail to Preserve Behavioral Fraud Patterns: A Benchmark on Temporal, Velocity, and Multi-Account Signals](https://mlsanity.com/paper/2604.13125/) (2026)
- [Evaluating quality in synthetic data generation for large tabular health datasets](https://mlsanity.com/paper/2604.15961/) (2026)
- [Self-Improving Tabular Language Models via Iterative Reward-Guided Post-Training](https://mlsanity.com/paper/2604.18966/) (2026)
- [On Privacy in Data-Space Tabular Diffusion Models: Influential Factors, Attacker Knowledge, and Metrics](https://mlsanity.com/paper/2605.06835/) (2026)
- [No Free Lunch for Synthetic Images under Data Scarcity Conditions](https://mlsanity.com/paper/2606.07640/) (2026)
- [Declarative Outcome-Conformant Synthesis: Exact, Closed-Form Specification Satisfaction and a Conformance Benchmark](https://mlsanity.com/paper/2606.08736/) (2026)
- [PSyGenTAB: A Privacy-Preserving Framework for Synthetic Clinical Tabular Data Generation via Constrained Optimization](https://mlsanity.com/paper/2606.18518/) (2026)
- [Seq2Synth: Benchmarking Temporal Fidelity in Synthetic Sequential Tabular Data](https://mlsanity.com/paper/2607.15606/) (2026)
- [K-IPO: Kendall-constrained Importance Preserving Oversampling for Imbalanced Tabular Data](https://mlsanity.com/paper/2607.16478/) (2026)
- [ProxyGuard: Direct Reliability Inference for Randomized Data Release Mechanisms with Shared Targets](https://mlsanity.com/paper/2608.18643/) (2026)
- [Beyond Marginals: A Multi-Dimensional Evaluation Framework for Multi-Table Synthetic Data Generation](https://mlsanity.com/paper/2610.06854/) (2026)

## Abstract (arXiv)

With the growing demand for synthetic data to address contemporary issues in machine learning, such as data scarcity, data fairness, and data privacy, having robust tools for assessing the utility and potential privacy risks of such data becomes crucial. SynthEval, a novel open-source evaluation framework distinguishes itself from existing tools by treating categorical and numerical attributes with equal care, without assuming any special kind of preprocessing steps. This~makes it applicable to virtually any synthetic dataset of tabular records. Our tool leverages statistical and machine learning techniques to comprehensively evaluate synthetic data fidelity and privacy-preserving integrity. SynthEval integrates a wide selection of metrics that can be used independently or in highly customisable benchmark configurations, and can easily be extended with additional metrics. In this paper, we describe SynthEval and illustrate its versatility with examples. The framework facilitates better benchmarking and more consistent comparisons of model capabilities.